2023-04-22 14:53:32,295 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1 2023-04-22 14:53:32,309 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-22 14:53:32,326 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-22 14:53:32,326 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a, deleteOnExit=true 2023-04-22 14:53:32,327 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-22 14:53:32,328 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/test.cache.data in system properties and HBase conf 2023-04-22 14:53:32,328 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/hadoop.tmp.dir in system properties and HBase conf 2023-04-22 14:53:32,329 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/hadoop.log.dir in system properties and HBase conf 2023-04-22 14:53:32,329 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-22 14:53:32,330 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-22 14:53:32,330 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-22 14:53:32,446 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-22 14:53:32,859 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-22 14:53:32,863 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-22 14:53:32,864 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-22 14:53:32,864 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-22 14:53:32,864 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 14:53:32,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-22 14:53:32,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-22 14:53:32,865 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 14:53:32,866 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 14:53:32,866 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-22 14:53:32,867 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/nfs.dump.dir in system properties and HBase conf 2023-04-22 14:53:32,867 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/java.io.tmpdir in system properties and HBase conf 2023-04-22 14:53:32,867 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 14:53:32,868 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-22 14:53:32,868 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-22 14:53:33,486 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 14:53:33,492 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 14:53:33,789 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-22 14:53:33,958 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-22 14:53:33,977 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:34,030 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:34,067 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/java.io.tmpdir/Jetty_localhost_39377_hdfs____vlew2f/webapp 2023-04-22 14:53:34,213 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39377 2023-04-22 14:53:34,271 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 14:53:34,271 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 14:53:34,725 WARN [Listener at localhost/40923] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:34,904 WARN [Listener at localhost/40923] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-22 14:53:34,925 WARN [Listener at localhost/40923] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:34,933 INFO [Listener at localhost/40923] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:34,940 INFO [Listener at localhost/40923] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/java.io.tmpdir/Jetty_localhost_33947_datanode____.jnk8vx/webapp 2023-04-22 14:53:35,036 INFO [Listener at localhost/40923] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33947 2023-04-22 14:53:35,385 WARN [Listener at localhost/45141] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:35,920 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1cd0337b926dde46: Processing first storage report for DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2 from datanode ea2500fb-007a-47ad-b293-b124c6c82704 2023-04-22 14:53:35,922 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1cd0337b926dde46: from storage DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2 node DatanodeRegistration(127.0.0.1:45837, datanodeUuid=ea2500fb-007a-47ad-b293-b124c6c82704, infoPort=33819, infoSecurePort=0, ipcPort=45141, storageInfo=lv=-57;cid=testClusterID;nsid=2110687604;c=1682175213564), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-22 14:53:35,923 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1cd0337b926dde46: Processing first storage report for DS-e5122947-fc97-4838-852b-7f3ebb8366bc from datanode ea2500fb-007a-47ad-b293-b124c6c82704 2023-04-22 14:53:35,923 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1cd0337b926dde46: from storage DS-e5122947-fc97-4838-852b-7f3ebb8366bc node DatanodeRegistration(127.0.0.1:45837, datanodeUuid=ea2500fb-007a-47ad-b293-b124c6c82704, infoPort=33819, infoSecurePort=0, ipcPort=45141, storageInfo=lv=-57;cid=testClusterID;nsid=2110687604;c=1682175213564), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,005 DEBUG [Listener at localhost/45141] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1 2023-04-22 14:53:36,073 INFO [Listener at localhost/45141] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a/zookeeper_0, clientPort=51853, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-22 14:53:36,089 INFO [Listener at localhost/45141] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51853 2023-04-22 14:53:36,102 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:36,104 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:36,770 INFO [Listener at localhost/45141] util.FSUtils(471): Created version file at hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c with version=8 2023-04-22 14:53:36,770 INFO [Listener at localhost/45141] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/hbase-staging 2023-04-22 14:53:37,083 INFO [Listener at localhost/45141] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-22 14:53:37,553 INFO [Listener at localhost/45141] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:37,584 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,585 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,585 INFO [Listener at localhost/45141] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:37,586 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,586 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:37,728 INFO [Listener at localhost/45141] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:37,806 DEBUG [Listener at localhost/45141] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-22 14:53:37,931 INFO [Listener at localhost/45141] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37547 2023-04-22 14:53:37,942 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:37,944 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:37,966 INFO [Listener at localhost/45141] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37547 connecting to ZooKeeper ensemble=127.0.0.1:51853 2023-04-22 14:53:38,006 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:375470x0, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37547-0x101ac5cb37f0000 connected 2023-04-22 14:53:38,041 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,042 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,046 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,054 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37547 2023-04-22 14:53:38,055 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37547 2023-04-22 14:53:38,055 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37547 2023-04-22 14:53:38,055 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37547 2023-04-22 14:53:38,056 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37547 2023-04-22 14:53:38,064 INFO [Listener at localhost/45141] master.HMaster(444): hbase.rootdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c, hbase.cluster.distributed=false 2023-04-22 14:53:38,141 INFO [Listener at localhost/45141] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:38,142 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,142 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,142 INFO [Listener at localhost/45141] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:38,142 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,142 INFO [Listener at localhost/45141] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:38,147 INFO [Listener at localhost/45141] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:38,150 INFO [Listener at localhost/45141] ipc.NettyRpcServer(120): Bind to /172.31.14.131:32935 2023-04-22 14:53:38,153 INFO [Listener at localhost/45141] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-22 14:53:38,160 DEBUG [Listener at localhost/45141] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-22 14:53:38,161 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,165 INFO [Listener at localhost/45141] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,167 INFO [Listener at localhost/45141] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:32935 connecting to ZooKeeper ensemble=127.0.0.1:51853 2023-04-22 14:53:38,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:329350x0, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:32935-0x101ac5cb37f0001 connected 2023-04-22 14:53:38,172 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,174 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,175 DEBUG [Listener at localhost/45141] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,175 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=32935 2023-04-22 14:53:38,176 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=32935 2023-04-22 14:53:38,180 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=32935 2023-04-22 14:53:38,184 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=32935 2023-04-22 14:53:38,187 DEBUG [Listener at localhost/45141] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=32935 2023-04-22 14:53:38,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:38,200 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 14:53:38,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:38,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,221 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:38,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 14:53:38,223 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37547,1682175216913 from backup master directory 2023-04-22 14:53:38,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 14:53:38,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:38,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 14:53:38,227 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:38,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:38,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-22 14:53:38,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-22 14:53:38,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/hbase.id with ID: 26aba3f2-fbe8-4aec-a205-3f39bf4aa7b5 2023-04-22 14:53:38,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:38,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2869d048 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:38,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b363ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:38,485 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:38,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-22 14:53:38,506 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-22 14:53:38,506 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-22 14:53:38,509 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:38,513 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:38,514 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:38,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store-tmp 2023-04-22 14:53:38,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:38,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 14:53:38,584 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:38,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:38,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 14:53:38,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:38,584 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:38,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:53:38,586 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/WALs/jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:38,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37547%2C1682175216913, suffix=, logDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/WALs/jenkins-hbase4.apache.org,37547,1682175216913, archiveDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/oldWALs, maxLogs=10 2023-04-22 14:53:38,668 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK] 2023-04-22 14:53:38,677 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:38,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/WALs/jenkins-hbase4.apache.org,37547,1682175216913/jenkins-hbase4.apache.org%2C37547%2C1682175216913.1682175218623 2023-04-22 14:53:38,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK]] 2023-04-22 14:53:38,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:38,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:38,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,754 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,821 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,830 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-22 14:53:38,857 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-22 14:53:38,873 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:38,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,883 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:38,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:38,912 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73834082, jitterRate=0.10021355748176575}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:53:38,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:53:38,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-22 14:53:38,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-22 14:53:38,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-22 14:53:38,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:38,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-22 14:53:38,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-04-22 14:53:38,979 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-22 14:53:39,006 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-22 14:53:39,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-22 14:53:39,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-22 14:53:39,042 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-22 14:53:39,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-22 14:53:39,052 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-22 14:53:39,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-22 14:53:39,061 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-22 14:53:39,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-22 14:53:39,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-22 14:53:39,080 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,080 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,080 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,082 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,37547,1682175216913, sessionid=0x101ac5cb37f0000, setting cluster-up flag (Was=false) 2023-04-22 14:53:39,097 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,104 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-22 14:53:39,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:39,110 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,116 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-22 14:53:39,118 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:39,121 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.hbase-snapshot/.tmp 2023-04-22 14:53:39,192 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(952): ClusterId : 26aba3f2-fbe8-4aec-a205-3f39bf4aa7b5 2023-04-22 14:53:39,198 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-22 14:53:39,206 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-22 14:53:39,207 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-22 14:53:39,212 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-22 14:53:39,213 DEBUG [RS:0;jenkins-hbase4:32935] zookeeper.ReadOnlyZKClient(139): Connect 0x15cedc5d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:39,225 DEBUG [RS:0;jenkins-hbase4:32935] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b093094, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:39,226 DEBUG [RS:0;jenkins-hbase4:32935] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f36b72d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:53:39,249 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:39,253 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:32935 2023-04-22 14:53:39,258 INFO [RS:0;jenkins-hbase4:32935] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-22 14:53:39,258 INFO [RS:0;jenkins-hbase4:32935] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-22 14:53:39,258 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1023): About to register with Master. 2023-04-22 14:53:39,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,262 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-22 14:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,263 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37547,1682175216913 with isa=jenkins-hbase4.apache.org/172.31.14.131:32935, startcode=1682175218140 2023-04-22 14:53:39,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175249275 2023-04-22 14:53:39,278 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-22 14:53:39,286 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:39,287 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-22 14:53:39,289 DEBUG [RS:0;jenkins-hbase4:32935] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-22 14:53:39,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-22 14:53:39,293 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:39,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-22 14:53:39,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-22 14:53:39,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-22 14:53:39,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-22 14:53:39,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-22 14:53:39,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-22 14:53:39,311 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-22 14:53:39,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-22 14:53:39,315 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-22 14:53:39,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219317,5,FailOnTimeoutGroup] 2023-04-22 14:53:39,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219329,5,FailOnTimeoutGroup] 2023-04-22 14:53:39,331 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-22 14:53:39,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,349 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:39,351 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:39,351 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c 2023-04-22 14:53:39,362 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46861, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-22 14:53:39,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,399 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c 2023-04-22 14:53:39,399 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40923 2023-04-22 14:53:39,399 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-22 14:53:39,404 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:53:39,405 DEBUG [RS:0;jenkins-hbase4:32935] zookeeper.ZKUtil(162): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,406 WARN [RS:0;jenkins-hbase4:32935] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:39,406 INFO [RS:0;jenkins-hbase4:32935] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:39,407 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/WALs/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,408 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,32935,1682175218140] 2023-04-22 14:53:39,420 DEBUG [RS:0;jenkins-hbase4:32935] zookeeper.ZKUtil(162): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,430 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-22 14:53:39,440 INFO [RS:0;jenkins-hbase4:32935] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-22 14:53:39,463 INFO [RS:0;jenkins-hbase4:32935] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-22 14:53:39,471 INFO [RS:0;jenkins-hbase4:32935] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-22 14:53:39,471 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,472 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-22 14:53:39,481 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,481 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,481 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,482 DEBUG [RS:0;jenkins-hbase4:32935] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,484 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,484 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,484 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,507 INFO [RS:0;jenkins-hbase4:32935] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-22 14:53:39,510 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,32935,1682175218140-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,534 INFO [RS:0;jenkins-hbase4:32935] regionserver.Replication(203): jenkins-hbase4.apache.org,32935,1682175218140 started 2023-04-22 14:53:39,534 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,32935,1682175218140, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:32935, sessionid=0x101ac5cb37f0001 2023-04-22 14:53:39,535 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-22 14:53:39,535 DEBUG [RS:0;jenkins-hbase4:32935] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,535 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,32935,1682175218140' 2023-04-22 14:53:39,535 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:39,536 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:39,536 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-22 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-22 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:32935] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,32935,1682175218140' 2023-04-22 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-22 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:32935] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-22 14:53:39,538 DEBUG [RS:0;jenkins-hbase4:32935] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-22 14:53:39,538 INFO [RS:0;jenkins-hbase4:32935] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-22 14:53:39,538 INFO [RS:0;jenkins-hbase4:32935] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-22 14:53:39,654 INFO [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C32935%2C1682175218140, suffix=, logDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/WALs/jenkins-hbase4.apache.org,32935,1682175218140, archiveDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/oldWALs, maxLogs=32 2023-04-22 14:53:39,674 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK] 2023-04-22 14:53:39,679 INFO [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/WALs/jenkins-hbase4.apache.org,32935,1682175218140/jenkins-hbase4.apache.org%2C32935%2C1682175218140.1682175219656 2023-04-22 14:53:39,679 DEBUG [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK]] 2023-04-22 14:53:39,771 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:39,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 14:53:39,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/info 2023-04-22 14:53:39,779 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 14:53:39,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 14:53:39,785 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:39,786 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 14:53:39,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 14:53:39,789 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/table 2023-04-22 14:53:39,790 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 14:53:39,791 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,792 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740 2023-04-22 14:53:39,793 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740 2023-04-22 14:53:39,797 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:53:39,799 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 14:53:39,803 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:39,805 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71858359, jitterRate=0.07077299058437347}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:53:39,805 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 14:53:39,805 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 14:53:39,806 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 14:53:39,806 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 14:53:39,806 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-22 14:53:39,806 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 14:53:39,807 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 14:53:39,808 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 14:53:39,813 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:39,813 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-22 14:53:39,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-22 14:53:39,840 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-22 14:53:39,851 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-22 14:53:40,003 DEBUG [jenkins-hbase4:37547] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-22 14:53:40,008 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,32935,1682175218140, state=OPENING 2023-04-22 14:53:40,018 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-22 14:53:40,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:40,022 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 14:53:40,027 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:53:40,221 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:40,223 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 14:53:40,227 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 14:53:40,241 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-22 14:53:40,241 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:40,246 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C32935%2C1682175218140.meta, suffix=.meta, logDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/WALs/jenkins-hbase4.apache.org,32935,1682175218140, archiveDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/oldWALs, maxLogs=32 2023-04-22 14:53:40,268 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK] 2023-04-22 14:53:40,281 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/WALs/jenkins-hbase4.apache.org,32935,1682175218140/jenkins-hbase4.apache.org%2C32935%2C1682175218140.meta.1682175220247.meta 2023-04-22 14:53:40,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45837,DS-f9df1d8f-4d8a-4742-b478-a8bff0b266d2,DISK]] 2023-04-22 14:53:40,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:40,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-22 14:53:40,306 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-22 14:53:40,313 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-22 14:53:40,319 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-22 14:53:40,319 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:40,320 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-22 14:53:40,320 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-22 14:53:40,324 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 14:53:40,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/info 2023-04-22 14:53:40,327 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/info 2023-04-22 14:53:40,327 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 14:53:40,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 14:53:40,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:40,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:40,331 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 14:53:40,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 14:53:40,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/table 2023-04-22 14:53:40,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/table 2023-04-22 14:53:40,336 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 14:53:40,337 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740 2023-04-22 14:53:40,342 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740 2023-04-22 14:53:40,346 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:53:40,349 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 14:53:40,351 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62155966, jitterRate=-0.07380393147468567}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:53:40,351 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 14:53:40,363 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682175220212 2023-04-22 14:53:40,383 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-22 14:53:40,384 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-22 14:53:40,384 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,32935,1682175218140, state=OPEN 2023-04-22 14:53:40,388 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-22 14:53:40,388 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 14:53:40,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-22 14:53:40,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,32935,1682175218140 in 361 msec 2023-04-22 14:53:40,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-22 14:53:40,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 570 msec 2023-04-22 14:53:40,402 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2320 sec 2023-04-22 14:53:40,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682175220402, completionTime=-1 2023-04-22 14:53:40,403 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-22 14:53:40,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:40,472 DEBUG [hconnection-0x7d5c3a4f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:40,476 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:40,496 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-22 14:53:40,496 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175280496 2023-04-22 14:53:40,496 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175340496 2023-04-22 14:53:40,496 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-04-22 14:53:40,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37547,1682175216913-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37547,1682175216913-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37547,1682175216913-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37547, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,531 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-22 14:53:40,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-22 14:53:40,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:40,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:40,558 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:40,561 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:40,591 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:40,594 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca empty. 2023-04-22 14:53:40,595 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:40,595 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-22 14:53:40,628 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:40,632 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8ad7b375dc1e6947072b970f80229aca, NAME => 'hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:53:40,650 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:40,650 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 8ad7b375dc1e6947072b970f80229aca, disabling compactions & flushes 2023-04-22 14:53:40,651 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:40,651 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:40,651 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. after waiting 0 ms 2023-04-22 14:53:40,651 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:40,651 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:40,651 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 8ad7b375dc1e6947072b970f80229aca: 2023-04-22 14:53:40,657 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:40,672 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175220660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175220660"}]},"ts":"1682175220660"} 2023-04-22 14:53:40,704 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:40,706 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:40,711 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175220706"}]},"ts":"1682175220706"} 2023-04-22 14:53:40,717 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-22 14:53:40,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8ad7b375dc1e6947072b970f80229aca, ASSIGN}] 2023-04-22 14:53:40,732 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8ad7b375dc1e6947072b970f80229aca, ASSIGN 2023-04-22 14:53:40,734 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=8ad7b375dc1e6947072b970f80229aca, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:53:40,885 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8ad7b375dc1e6947072b970f80229aca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:40,885 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175220884"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175220884"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175220884"}]},"ts":"1682175220884"} 2023-04-22 14:53:40,892 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 8ad7b375dc1e6947072b970f80229aca, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:53:41,054 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:41,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8ad7b375dc1e6947072b970f80229aca, NAME => 'hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:41,055 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,056 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:41,056 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,056 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,058 INFO [StoreOpener-8ad7b375dc1e6947072b970f80229aca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,061 DEBUG [StoreOpener-8ad7b375dc1e6947072b970f80229aca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/info 2023-04-22 14:53:41,061 DEBUG [StoreOpener-8ad7b375dc1e6947072b970f80229aca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/info 2023-04-22 14:53:41,061 INFO [StoreOpener-8ad7b375dc1e6947072b970f80229aca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ad7b375dc1e6947072b970f80229aca columnFamilyName info 2023-04-22 14:53:41,062 INFO [StoreOpener-8ad7b375dc1e6947072b970f80229aca-1] regionserver.HStore(310): Store=8ad7b375dc1e6947072b970f80229aca/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:41,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:53:41,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:41,075 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8ad7b375dc1e6947072b970f80229aca; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70106921, jitterRate=0.04467453062534332}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:53:41,075 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8ad7b375dc1e6947072b970f80229aca: 2023-04-22 14:53:41,077 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca., pid=6, masterSystemTime=1682175221047 2023-04-22 14:53:41,081 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:41,081 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:53:41,083 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8ad7b375dc1e6947072b970f80229aca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:41,083 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175221082"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175221082"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175221082"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175221082"}]},"ts":"1682175221082"} 2023-04-22 14:53:41,091 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-22 14:53:41,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 8ad7b375dc1e6947072b970f80229aca, server=jenkins-hbase4.apache.org,32935,1682175218140 in 195 msec 2023-04-22 14:53:41,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-22 14:53:41,096 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=8ad7b375dc1e6947072b970f80229aca, ASSIGN in 364 msec 2023-04-22 14:53:41,097 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:41,098 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175221097"}]},"ts":"1682175221097"} 2023-04-22 14:53:41,103 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-22 14:53:41,107 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:41,110 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 563 msec 2023-04-22 14:53:41,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-22 14:53:41,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:41,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:41,222 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,228 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 37 msec 2023-04-22 14:53:41,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:41,275 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 97 msec 2023-04-22 14:53:41,352 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-22 14:53:41,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-22 14:53:41,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.140sec 2023-04-22 14:53:41,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-22 14:53:41,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-22 14:53:41,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-22 14:53:41,372 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37547,1682175216913-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-22 14:53:41,373 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37547,1682175216913-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-22 14:53:41,385 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-22 14:53:41,398 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:41,403 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:41,417 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:41,435 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:41,448 INFO [Listener at localhost/45141] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:53:41,478 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=172, AvailableMemoryMB=13333 2023-04-22 14:53:41,496 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:53:41,502 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:53:41,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:53:41,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:41,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:53:41,522 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:41,525 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:41,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-22 14:53:41,528 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:41,529 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 empty. 2023-04-22 14:53:41,531 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:41,531 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:53:41,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:41,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:41,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:41,950 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:41,952 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 58845d2ad95f1e691d0bdc0dfabebf52, NAME => 'TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:53:42,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:42,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:42,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 58845d2ad95f1e691d0bdc0dfabebf52, disabling compactions & flushes 2023-04-22 14:53:42,367 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. after waiting 0 ms 2023-04-22 14:53:42,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,367 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:42,372 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:42,374 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175222373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175222373"}]},"ts":"1682175222373"} 2023-04-22 14:53:42,377 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:42,378 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:42,379 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175222378"}]},"ts":"1682175222378"} 2023-04-22 14:53:42,381 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:53:42,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, ASSIGN}] 2023-04-22 14:53:42,388 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, ASSIGN 2023-04-22 14:53:42,390 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:53:42,541 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=58845d2ad95f1e691d0bdc0dfabebf52, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:42,542 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175222541"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175222541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175222541"}]},"ts":"1682175222541"} 2023-04-22 14:53:42,546 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:53:42,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:42,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 58845d2ad95f1e691d0bdc0dfabebf52, NAME => 'TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:42,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:42,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,711 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,717 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A 2023-04-22 14:53:42,717 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A 2023-04-22 14:53:42,719 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:53:42,719 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 58845d2ad95f1e691d0bdc0dfabebf52 columnFamilyName A 2023-04-22 14:53:42,721 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(310): Store=58845d2ad95f1e691d0bdc0dfabebf52/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:42,721 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,723 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B 2023-04-22 14:53:42,723 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B 2023-04-22 14:53:42,724 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:53:42,724 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 58845d2ad95f1e691d0bdc0dfabebf52 columnFamilyName B 2023-04-22 14:53:42,725 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(310): Store=58845d2ad95f1e691d0bdc0dfabebf52/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:42,726 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,728 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C 2023-04-22 14:53:42,728 DEBUG [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C 2023-04-22 14:53:42,729 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:53:42,729 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 58845d2ad95f1e691d0bdc0dfabebf52 columnFamilyName C 2023-04-22 14:53:42,730 INFO [StoreOpener-58845d2ad95f1e691d0bdc0dfabebf52-1] regionserver.HStore(310): Store=58845d2ad95f1e691d0bdc0dfabebf52/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:42,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:53:42,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:42,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:42,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 58845d2ad95f1e691d0bdc0dfabebf52; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74147040, jitterRate=0.10487699508666992}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:53:42,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:42,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., pid=11, masterSystemTime=1682175222699 2023-04-22 14:53:42,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:42,754 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=58845d2ad95f1e691d0bdc0dfabebf52, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:42,754 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175222754"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175222754"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175222754"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175222754"}]},"ts":"1682175222754"} 2023-04-22 14:53:42,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-22 14:53:42,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 in 212 msec 2023-04-22 14:53:42,767 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-22 14:53:42,767 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, ASSIGN in 377 msec 2023-04-22 14:53:42,769 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:42,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175222769"}]},"ts":"1682175222769"} 2023-04-22 14:53:42,772 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:53:42,776 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:42,779 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2590 sec 2023-04-22 14:53:43,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-22 14:53:43,653 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-22 14:53:43,659 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,667 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,671 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,681 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,685 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:53:43,688 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:53:43,697 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,702 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,703 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,709 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,710 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,714 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,715 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,720 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,721 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,726 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,727 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,737 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,737 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,772 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,773 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,777 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,777 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:43,781 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:43,788 DEBUG [hconnection-0x2a81f259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,789 DEBUG [hconnection-0x74b74e95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,790 DEBUG [hconnection-0x7c8e030b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,792 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,794 DEBUG [hconnection-0x6d40587f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,794 DEBUG [hconnection-0x1e045d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,794 DEBUG [hconnection-0x3bac190f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,794 DEBUG [hconnection-0x26f80c4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,795 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,796 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60622, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,798 DEBUG [hconnection-0x1bd47f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:43,798 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,804 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,805 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,805 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,806 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,809 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:43,812 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:43,831 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:43,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:43,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:43,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:43,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:43,837 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:43,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:43,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:43,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:43,840 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:43,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:43,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:43,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:43,848 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:43,849 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:43,853 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 14:53:43,917 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:43,918 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:43,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:43,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:43,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:43,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:43,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:43,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:44,028 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/bdc9177ebe524bde99fb1082488ac204 2023-04-22 14:53:44,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224019, max timestamp=1682175224046Num uniques 859; 2023-04-22 14:53:44,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224019, max timestamp=1682175224046Num uniques 859; 2023-04-22 14:53:44,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224019, max timestamp=1682175224046Num uniques 859; 2023-04-22 14:53:44,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,190 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/ec76bcd7165e4b66878284463df541fa 2023-04-22 14:53:44,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175224211, max timestamp=1682175224247Num uniques 1018; 2023-04-22 14:53:44,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:53:44,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175224211, max timestamp=1682175224247Num uniques 1018; 2023-04-22 14:53:44,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175224211, max timestamp=1682175224247Num uniques 1018; 2023-04-22 14:53:44,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-22 14:53:44,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-22 14:53:44,283 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/72e4798ba4ef4e90a1ca84c8bd96b2b5 2023-04-22 14:53:44,297 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/bdc9177ebe524bde99fb1082488ac204 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204 2023-04-22 14:53:44,312 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204, entries=100, sequenceid=8, filesize=9.3 K 2023-04-22 14:53:44,326 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/ec76bcd7165e4b66878284463df541fa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa 2023-04-22 14:53:44,350 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa, entries=100, sequenceid=8, filesize=9.3 K 2023-04-22 14:53:44,355 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/72e4798ba4ef4e90a1ca84c8bd96b2b5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5 2023-04-22 14:53:44,405 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5, entries=100, sequenceid=8, filesize=9.3 K 2023-04-22 14:53:44,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175224385, max timestamp=1682175224395Num uniques 1027; 2023-04-22 14:53:44,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:53:44,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175224385, max timestamp=1682175224395Num uniques 1009; 2023-04-22 14:53:44,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-22 14:53:44,426 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=70.76 KB/72462 for 58845d2ad95f1e691d0bdc0dfabebf52 in 572ms, sequenceid=8, compaction requested=false 2023-04-22 14:53:44,428 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 14:53:44,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175224385, max timestamp=1682175224397Num uniques 1009; 2023-04-22 14:53:44,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-22 14:53:44,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:44,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=53.67 KB heapSize=144.23 KB 2023-04-22 14:53:44,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:44,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:44,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:44,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:44,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:44,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:44,435 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:44,441 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:44,443 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,446 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:44,447 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,447 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:44,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:44,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73222e4e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:44,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:44,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,467 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:44,474 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:44,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:44,477 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:44,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:44,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:44,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:44,477 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:44,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:44,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:44,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:44,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,531 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:44,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/158d7031a8fc472f84e6f425d813d0f5 2023-04-22 14:53:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:44,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:44,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:44,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:44,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,555 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:44,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:44,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:44,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:44,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:44,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224535, max timestamp=1682175224586Num uniques 859; 2023-04-22 14:53:44,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224535, max timestamp=1682175224586Num uniques 859; 2023-04-22 14:53:44,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175224535, max timestamp=1682175224586Num uniques 859; 2023-04-22 14:53:44,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:44,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/086dd856175843be9125d2a8bde5d590 2023-04-22 14:53:44,684 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:44,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175224668, max timestamp=1682175224692Num uniques 1018; 2023-04-22 14:53:44,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:44,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175224668, max timestamp=1682175224692Num uniques 1018; 2023-04-22 14:53:44,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:44,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175224668, max timestamp=1682175224692Num uniques 1018; 2023-04-22 14:53:44,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:44,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:44,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:44,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:44,748 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:44,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:44,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:44,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:44,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,757 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:44,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:44,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:44,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:44,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:44,782 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:44,782 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:44,783 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:44,783 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:44,783 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:44,783 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:44,793 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,793 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:44,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c052dc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:44,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:44,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,799 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:44,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:44,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:44,811 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:44,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:44,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:44,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:44,811 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:44,817 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:44,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:44,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:44,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,823 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:44,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:44,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:44,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:44,833 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,833 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:44,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:44,833 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:44,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:44,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:44,834 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:44,834 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:44,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175284826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:44,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175284834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175284863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175284863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175284865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,934 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:44,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:44,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:44,961 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:44,961 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:44,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:44,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:44,964 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,964 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,966 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:44,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:44,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:44,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:44,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:44,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:44,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:44,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:44,989 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:44,989 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:44,989 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:44,990 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:44,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:45,001 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:45,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:45,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608e900[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:45,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:45,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175285002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:45,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:45,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175285007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175285009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175285009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175285009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,018 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:45,036 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:45,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:45,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:45,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:45,036 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:45,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:45,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:45,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:45,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,065 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:45,065 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:45,065 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:45,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:45,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:45,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:45,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:45,069 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:45,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:45,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:45,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:45,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7815d728763e46fdb24921578b8d5e31 2023-04-22 14:53:45,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/158d7031a8fc472f84e6f425d813d0f5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5 2023-04-22 14:53:45,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5, entries=400, sequenceid=67, filesize=23.0 K 2023-04-22 14:53:45,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/086dd856175843be9125d2a8bde5d590 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590 2023-04-22 14:53:45,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:45,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:45,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590, entries=150, sequenceid=67, filesize=11.6 K 2023-04-22 14:53:45,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:45,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7815d728763e46fdb24921578b8d5e31 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31 2023-04-22 14:53:45,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31, entries=150, sequenceid=67, filesize=11.6 K 2023-04-22 14:53:45,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~144.19 KB/147648, currentSize=134.18 KB/137400 for 58845d2ad95f1e691d0bdc0dfabebf52 in 759ms, sequenceid=67, compaction requested=false 2023-04-22 14:53:45,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:45,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:45,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:45,201 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:45,201 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:45,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:45,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:45,203 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,203 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:45,208 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:45,208 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:45,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:45,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:45,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:45,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:45,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:45,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:45,216 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:45,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:45,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:45,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:45,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175285233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175285232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175285235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175285238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175285239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,287 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/11b5d62e96674c1e95eecac589fbe39a 2023-04-22 14:53:45,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175285346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175285346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175285347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175285347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175285347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175285549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175285562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175285568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175285569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175285573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,752 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/e78d168a67cd4b698dec50dda7a7d557 2023-04-22 14:53:45,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175285857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175285865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175285875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175285881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175285881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:45,905 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:53:45,976 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-22 14:53:45,978 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-22 14:53:46,213 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6abc8233737940d8a92df884361ee072 2023-04-22 14:53:46,237 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/11b5d62e96674c1e95eecac589fbe39a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a 2023-04-22 14:53:46,254 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a, entries=150, sequenceid=120, filesize=11.6 K 2023-04-22 14:53:46,260 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/e78d168a67cd4b698dec50dda7a7d557 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557 2023-04-22 14:53:46,278 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557, entries=150, sequenceid=120, filesize=11.6 K 2023-04-22 14:53:46,297 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6abc8233737940d8a92df884361ee072 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072 2023-04-22 14:53:46,315 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072, entries=150, sequenceid=120, filesize=11.6 K 2023-04-22 14:53:46,317 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=53.67 KB/54960 for 58845d2ad95f1e691d0bdc0dfabebf52 in 1101ms, sequenceid=120, compaction requested=true 2023-04-22 14:53:46,318 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:46,321 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,321 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:46,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:46,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:46,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:46,326 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d67fb1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:46,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:46,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,350 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:46,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:46,354 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:46,365 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:46,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:46,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:46,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:46,365 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:46,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,368 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:46,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:53:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:46,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:46,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,397 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,397 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,400 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:46,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,401 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,401 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,401 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:46,401 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:46,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175286466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175286468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175286469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175286471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175286472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,502 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:46,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:46,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:46,544 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:46,544 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,546 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:46,546 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:46,546 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,550 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:46,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:46,560 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:46,560 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:46,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:46,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:46,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,566 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e4755b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:46,570 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:46,570 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,571 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,572 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,572 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:46,575 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:46,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175286583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175286583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175286584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175286588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175286589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:46,599 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:46,599 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:46,599 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:46,599 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:46,599 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:46,603 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,603 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,603 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,621 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:46,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:46,625 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:46,625 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:46,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,725 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:46,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:46,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:46,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:46,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:46,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:46,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:46,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:46,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:46,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:46,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:46,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,763 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34439989[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:46,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:46,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:46,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:46,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:46,774 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:46,780 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:46,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:46,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:46,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:46,780 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:46,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,784 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175286789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175286791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175286791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175286791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:46,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175286798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,813 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:46,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:46,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:46,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:46,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/ac4e52037e54418f9a5c870aacd45b84 2023-04-22 14:53:46,917 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:46,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/7c4e0f9f1f6140daacc695f95672260f 2023-04-22 14:53:46,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:46,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:46,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:46,933 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:46,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:46,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:46,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:46,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:46,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:46,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:46,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:46,953 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,953 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:46,953 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:46,953 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:46,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:46,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:46,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:46,956 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,956 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754226e8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:46,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:46,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:46,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:46,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:46,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:46,978 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:46,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:46,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:46,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:46,978 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:46,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:46,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:46,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/092da0137d0440e18bd331da5055826e 2023-04-22 14:53:47,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:47,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:47,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:47,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:47,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/ac4e52037e54418f9a5c870aacd45b84 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84 2023-04-22 14:53:47,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84, entries=250, sequenceid=133, filesize=16.4 K 2023-04-22 14:53:47,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/7c4e0f9f1f6140daacc695f95672260f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f 2023-04-22 14:53:47,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f, entries=150, sequenceid=133, filesize=11.8 K 2023-04-22 14:53:47,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/092da0137d0440e18bd331da5055826e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e 2023-04-22 14:53:47,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e, entries=150, sequenceid=133, filesize=11.8 K 2023-04-22 14:53:47,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 58845d2ad95f1e691d0bdc0dfabebf52 in 685ms, sequenceid=133, compaction requested=true 2023-04-22 14:53:47,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:47,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:47,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:47,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:47,066 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:47,066 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:47,074 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:47,077 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:47,077 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,078 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=44.3 K 2023-04-22 14:53:47,079 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 14:53:47,080 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-22 14:53:47,080 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ec76bcd7165e4b66878284463df541fa, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682175223822 2023-04-22 14:53:47,085 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 086dd856175843be9125d2a8bde5d590, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682175224391 2023-04-22 14:53:47,086 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e78d168a67cd4b698dec50dda7a7d557, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682175224817 2023-04-22 14:53:47,087 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7c4e0f9f1f6140daacc695f95672260f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225235 2023-04-22 14:53:47,090 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 61784 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:47,090 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:47,090 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,091 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=60.3 K 2023-04-22 14:53:47,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:47,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:47,109 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting bdc9177ebe524bde99fb1082488ac204, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682175223822 2023-04-22 14:53:47,110 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 158d7031a8fc472f84e6f425d813d0f5, keycount=400, bloomtype=ROW, size=23.0 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682175224385 2023-04-22 14:53:47,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175226449, max timestamp=1682175227105Num uniques 859; 2023-04-22 14:53:47,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:47,111 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 11b5d62e96674c1e95eecac589fbe39a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682175224817 2023-04-22 14:53:47,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:47,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175226449, max timestamp=1682175227105Num uniques 859; 2023-04-22 14:53:47,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:47,114 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ac4e52037e54418f9a5c870aacd45b84, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225228 2023-04-22 14:53:47,117 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-22 14:53:47,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:47,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:47,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:47,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:47,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:47,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:47,127 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:47,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:47,152 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#13 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:47,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:47,155 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,155 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,172 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#14 average throughput is 0.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:47,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:47,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:47,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:47,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:47,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:47,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:47,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:47,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:47,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:47,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:47,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fde406[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:47,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:47,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,266 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:47,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:47,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:47,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:47,274 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:47,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:47,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:47,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:47,274 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:47,278 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,278 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,278 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,279 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,281 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,289 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:47,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,289 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:47,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:47,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:47,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175287228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175287250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175227212, max timestamp=1682175227229Num uniques 859; 2023-04-22 14:53:47,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175227212, max timestamp=1682175227247Num uniques 859; 2023-04-22 14:53:47,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 14:53:47,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175227212, max timestamp=1682175227247Num uniques 859; 2023-04-22 14:53:47,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 14:53:47,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-22 14:53:47,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee 2023-04-22 14:53:47,378 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/b08146ea20b24bae96a6b62b1e8a3a35 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b08146ea20b24bae96a6b62b1e8a3a35 2023-04-22 14:53:47,393 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:47,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:47,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:47,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175287382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:47,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:47,404 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:47,404 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,408 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:47,408 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:47,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175287402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175287405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,414 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,417 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,417 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,417 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/46057398faf0432eb4a2d6f124a59fff 2023-04-22 14:53:47,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:47,420 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into b08146ea20b24bae96a6b62b1e8a3a35(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:47,420 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:47,420 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=12, startTime=1682175227065; duration=0sec 2023-04-22 14:53:47,421 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:47,421 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:47,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:47,429 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:47,431 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:47,431 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:47,433 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,433 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:47,433 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:47,433 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:47,435 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:47,435 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:47,436 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,436 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=44.3 K 2023-04-22 14:53:47,437 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,437 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 72e4798ba4ef4e90a1ca84c8bd96b2b5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682175223822 2023-04-22 14:53:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:47,445 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7815d728763e46fdb24921578b8d5e31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682175224391 2023-04-22 14:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175287418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175287418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,448 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6abc8233737940d8a92df884361ee072, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1682175224817 2023-04-22 14:53:47,457 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 092da0137d0440e18bd331da5055826e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225235 2023-04-22 14:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d50bcf[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:47,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:47,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,465 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:47,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:47,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:47,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:47,468 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,469 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:47,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:47,480 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:47,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:47,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:47,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:47,480 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:47,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,483 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,495 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,497 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:47,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,501 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#17 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:47,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175287504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175287516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175287517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,569 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6963ddc56a384e23840da85e13294d80 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6963ddc56a384e23840da85e13294d80 2023-04-22 14:53:47,593 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into 6963ddc56a384e23840da85e13294d80(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:47,593 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:47,593 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=12, startTime=1682175227066; duration=0sec 2023-04-22 14:53:47,595 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:47,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:47,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:47,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:47,605 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:47,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,607 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:47,607 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:47,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,609 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:47,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:47,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:47,628 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,628 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:47,628 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:47,628 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:47,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:47,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:47,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:47,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff2556d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:47,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:47,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:47,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:47,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:47,653 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:47,653 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:47,653 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:47,653 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:47,653 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:47,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175287652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,657 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175287654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,659 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,663 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,663 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,663 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:47,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:47,663 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:47,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:47,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:47,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175287717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175287727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175287727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,760 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8e3f2f292f65440a96f6be279153718b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8e3f2f292f65440a96f6be279153718b 2023-04-22 14:53:47,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:47,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:47,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:47,775 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into 8e3f2f292f65440a96f6be279153718b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:47,776 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:47,776 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=12, startTime=1682175227066; duration=0sec 2023-04-22 14:53:47,776 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:47,776 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:47,777 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,781 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:47,781 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:47,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,785 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:47,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:47,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:47,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:47,793 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:47,796 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:47,798 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b63531d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:47,801 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:47,802 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:47,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:47,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:47,809 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:47,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:47,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:47,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:47,810 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:47,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,814 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,820 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,820 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,820 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,821 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,821 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:47,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:47,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,822 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:47,823 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:47,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:47,923 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:47,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:47,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:47,933 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:47,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:47,939 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:47,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:47,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:47,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:47,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/1b3527920b68433786760cc0baed2be8 2023-04-22 14:53:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:47,964 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,964 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:47,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,964 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175287959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,964 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:47,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175287960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee 2023-04-22 14:53:47,971 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453053a3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:47,975 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:47,975 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,977 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:47,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:47,988 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:47,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee, entries=200, sequenceid=154, filesize=14.1 K 2023-04-22 14:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:47,996 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:47,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:47,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:47,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:47,996 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:47,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/46057398faf0432eb4a2d6f124a59fff as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff 2023-04-22 14:53:47,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:47,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:48,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,002 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:48,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:48,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:48,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:48,012 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:48,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:48,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:48,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff, entries=150, sequenceid=154, filesize=11.8 K 2023-04-22 14:53:48,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/1b3527920b68433786760cc0baed2be8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8 2023-04-22 14:53:48,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175288027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8, entries=150, sequenceid=154, filesize=11.8 K 2023-04-22 14:53:48,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=132.99 KB/136185 for 58845d2ad95f1e691d0bdc0dfabebf52 in 940ms, sequenceid=154, compaction requested=false 2023-04-22 14:53:48,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:48,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:48,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=139.70 KB heapSize=402.87 KB 2023-04-22 14:53:48,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:48,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175227380, max timestamp=1682175228041Num uniques 1009; 2023-04-22 14:53:48,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175227380, max timestamp=1682175228041Num uniques 1009; 2023-04-22 14:53:48,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-22 14:53:48,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-22 14:53:48,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175227380, max timestamp=1682175228041Num uniques 1009; 2023-04-22 14:53:48,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-22 14:53:48,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:48,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:48,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:48,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:48,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/98d90cf740654631895c9498b8630251 2023-04-22 14:53:48,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:48,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:48,135 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:48,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:48,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:48,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:48,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,139 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:48,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:48,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:48,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:48,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:48,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:48,166 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,168 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a14b3b9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:48,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:48,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,173 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:48,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:48,180 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:48,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:48,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:48,181 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:48,181 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:48,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:48,183 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,185 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:48,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:48,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:48,190 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,190 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,190 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:48,190 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:48,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:48,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:48,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:48,194 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:48,194 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:48,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175228104, max timestamp=1682175228167Num uniques 859; 2023-04-22 14:53:48,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:48,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175228104, max timestamp=1682175228167Num uniques 859; 2023-04-22 14:53:48,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175228104, max timestamp=1682175228167Num uniques 859; 2023-04-22 14:53:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:48,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:48,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/b5d1e330e6e54365848985e0a720aa18 2023-04-22 14:53:48,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/60a4eebc923c4d0db09d1bb27f324032 2023-04-22 14:53:48,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/98d90cf740654631895c9498b8630251 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251 2023-04-22 14:53:48,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 14:53:48,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/b5d1e330e6e54365848985e0a720aa18 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18 2023-04-22 14:53:48,297 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:48,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:48,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175228269, max timestamp=1682175228306Num uniques 1018; 2023-04-22 14:53:48,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:48,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175228269, max timestamp=1682175228306Num uniques 1018; 2023-04-22 14:53:48,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:48,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175228269, max timestamp=1682175228306Num uniques 1018; 2023-04-22 14:53:48,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:48,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:48,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:48,315 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:48,315 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:48,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:48,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:48,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 14:53:48,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:48,319 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:48,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:48,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/60a4eebc923c4d0db09d1bb27f324032 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032 2023-04-22 14:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:48,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:48,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032, entries=150, sequenceid=196, filesize=11.8 K 2023-04-22 14:53:48,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=60.38 KB/61830 for 58845d2ad95f1e691d0bdc0dfabebf52 in 273ms, sequenceid=196, compaction requested=true 2023-04-22 14:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:48,331 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:48,331 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:48,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:48,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:48,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:48,333 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:48,333 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:48,346 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,347 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:48,347 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=60.38 KB heapSize=160.31 KB 2023-04-22 14:53:48,347 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:48,347 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:48,347 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:48,348 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,348 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,348 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:48,348 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,348 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8e3f2f292f65440a96f6be279153718b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=35.4 K 2023-04-22 14:53:48,348 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:48,349 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:48,349 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,349 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b08146ea20b24bae96a6b62b1e8a3a35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=37.7 K 2023-04-22 14:53:48,349 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8e3f2f292f65440a96f6be279153718b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225235 2023-04-22 14:53:48,348 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:48,352 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting b08146ea20b24bae96a6b62b1e8a3a35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225235 2023-04-22 14:53:48,352 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 46057398faf0432eb4a2d6f124a59fff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175226466 2023-04-22 14:53:48,353 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:48,354 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 2b4e9b4f9e7e43f6a1871d2ad05f26ee, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175226449 2023-04-22 14:53:48,354 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b5d1e330e6e54365848985e0a720aa18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:48,355 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 98d90cf740654631895c9498b8630251, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:48,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:48,386 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:48,409 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#23 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:48,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175228409, max timestamp=1682175228446Num uniques 859; 2023-04-22 14:53:48,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:53:48,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175228409, max timestamp=1682175228446Num uniques 859; 2023-04-22 14:53:48,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:48,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175228409, max timestamp=1682175228446Num uniques 859; 2023-04-22 14:53:48,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:48,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175288499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175288500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175288505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175288505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175288557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175288611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175288612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175288614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175288614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175288820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175288822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175288822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:48,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175288823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:48,871 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/8ba6214828d74071861f9d22c6c85f6f 2023-04-22 14:53:48,873 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/45e24fe447fd41f8a110c730c318cabe as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/45e24fe447fd41f8a110c730c318cabe 2023-04-22 14:53:48,887 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into 45e24fe447fd41f8a110c730c318cabe(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:48,887 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:48,889 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=13, startTime=1682175228331; duration=0sec 2023-04-22 14:53:48,889 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:48,889 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:48,892 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:48,892 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:48,892 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:48,892 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6963ddc56a384e23840da85e13294d80, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=35.4 K 2023-04-22 14:53:48,893 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6963ddc56a384e23840da85e13294d80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682175225235 2023-04-22 14:53:48,894 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1b3527920b68433786760cc0baed2be8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175226466 2023-04-22 14:53:48,895 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 60a4eebc923c4d0db09d1bb27f324032, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:48,905 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/d6ea0915a6a64ece980e61d4e158115e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d6ea0915a6a64ece980e61d4e158115e 2023-04-22 14:53:48,925 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into d6ea0915a6a64ece980e61d4e158115e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:48,925 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:48,925 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=13, startTime=1682175228331; duration=0sec 2023-04-22 14:53:48,926 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:48,939 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#25 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:49,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175289125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175289131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175289135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175289138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,363 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/0cb446f56b044f34b7c61edfa4306aed 2023-04-22 14:53:49,402 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/c483d59d8c8a4d98b48549ad1b623e1c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c483d59d8c8a4d98b48549ad1b623e1c 2023-04-22 14:53:49,420 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/92d36ac484ee439d96166340862e29f9 2023-04-22 14:53:49,423 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into c483d59d8c8a4d98b48549ad1b623e1c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:49,424 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:49,424 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=13, startTime=1682175228332; duration=0sec 2023-04-22 14:53:49,424 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:49,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/8ba6214828d74071861f9d22c6c85f6f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f 2023-04-22 14:53:49,476 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f, entries=150, sequenceid=238, filesize=11.8 K 2023-04-22 14:53:49,478 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/0cb446f56b044f34b7c61edfa4306aed as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed 2023-04-22 14:53:49,495 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed, entries=150, sequenceid=238, filesize=11.8 K 2023-04-22 14:53:49,499 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/92d36ac484ee439d96166340862e29f9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9 2023-04-22 14:53:49,507 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9, entries=150, sequenceid=238, filesize=11.8 K 2023-04-22 14:53:49,517 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~160.27 KB/164112, currentSize=131.94 KB/135110 for 58845d2ad95f1e691d0bdc0dfabebf52 in 1170ms, sequenceid=238, compaction requested=false 2023-04-22 14:53:49,518 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:49,518 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:49,522 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5770daab[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,525 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:49,525 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,527 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:49,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,541 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:49,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:49,542 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,543 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,543 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,543 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,545 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,548 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,553 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:49,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:49,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,555 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,555 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:49,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-22 14:53:49,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:49,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:49,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:49,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:49,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:49,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:49,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175289594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/e1c92eb4a92041c397eaedc503d5e403 2023-04-22 14:53:49,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175289631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175289641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175289641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175289643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,656 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:49,667 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:49,667 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,669 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:49,669 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:49,669 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,669 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,671 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:49,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:49,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:49,673 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:49,673 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:49,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da90e3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,688 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:49,688 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175289696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,698 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:49,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:49,701 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,707 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:49,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:49,707 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:49,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,720 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,721 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:49,831 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:49,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:49,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:49,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,837 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,838 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,838 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:49,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,840 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:49,842 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:49,842 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:49,842 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:49,843 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,845 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413076d0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:49,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,852 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:49,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:49,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:49,856 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,860 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:49,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:49,860 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,863 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,863 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,864 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,875 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:49,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:49,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,878 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,878 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,878 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,878 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175289900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,978 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:49,990 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:49,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:49,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:49,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:49,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:49,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:50,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:50,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,004 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,004 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:50,004 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:50,004 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:50,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e768c34[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:50,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,016 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:50,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:50,020 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,024 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:50,024 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,024 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,024 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:50,024 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,027 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:50,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,041 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,041 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/192624ff5c544e61b1c9bbe5edc81d20 2023-04-22 14:53:50,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/c540700feafd418c8876d078ad2c2de1 2023-04-22 14:53:50,141 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:50,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/e1c92eb4a92041c397eaedc503d5e403 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403 2023-04-22 14:53:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:50,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:50,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:50,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:50,148 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:50,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:50,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:50,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403, entries=200, sequenceid=278, filesize=14.3 K 2023-04-22 14:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:50,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:50,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/192624ff5c544e61b1c9bbe5edc81d20 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20 2023-04-22 14:53:50,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,172 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,172 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:50,172 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:50,172 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:50,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f02bad3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:50,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,180 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:50,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:50,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20, entries=150, sequenceid=278, filesize=11.9 K 2023-04-22 14:53:50,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/c540700feafd418c8876d078ad2c2de1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1 2023-04-22 14:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,193 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:50,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:50,193 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,208 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,208 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,208 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,209 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1, entries=150, sequenceid=278, filesize=11.9 K 2023-04-22 14:53:50,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175290206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 58845d2ad95f1e691d0bdc0dfabebf52 in 659ms, sequenceid=278, compaction requested=true 2023-04-22 14:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:50,230 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:50,230 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:50,236 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38938 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:50,236 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:50,236 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,237 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/45e24fe447fd41f8a110c730c318cabe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=38.0 K 2023-04-22 14:53:50,237 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 45e24fe447fd41f8a110c730c318cabe, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:50,238 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:50,238 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:50,239 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,239 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d6ea0915a6a64ece980e61d4e158115e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=35.6 K 2023-04-22 14:53:50,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,240 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8ba6214828d74071861f9d22c6c85f6f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682175228314 2023-04-22 14:53:50,241 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d6ea0915a6a64ece980e61d4e158115e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:50,243 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e1c92eb4a92041c397eaedc503d5e403, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228493 2023-04-22 14:53:50,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,244 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0cb446f56b044f34b7c61edfa4306aed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682175228314 2023-04-22 14:53:50,245 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 192624ff5c544e61b1c9bbe5edc81d20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228496 2023-04-22 14:53:50,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:50,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:50,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,249 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:50,266 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:50,274 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:50,314 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/9ed1ddaa25c943ed887d47f3ef0cdf89 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/9ed1ddaa25c943ed887d47f3ef0cdf89 2023-04-22 14:53:50,332 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/0bdd4b943adf42f6acb35fb75c96903d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0bdd4b943adf42f6acb35fb75c96903d 2023-04-22 14:53:50,344 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into 0bdd4b943adf42f6acb35fb75c96903d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:50,344 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:50,344 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=13, startTime=1682175230230; duration=0sec 2023-04-22 14:53:50,345 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:50,345 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:50,345 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into 9ed1ddaa25c943ed887d47f3ef0cdf89(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:50,345 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:50,345 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=13, startTime=1682175230230; duration=0sec 2023-04-22 14:53:50,345 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:50,346 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:50,347 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:50,347 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,347 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c483d59d8c8a4d98b48549ad1b623e1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=35.6 K 2023-04-22 14:53:50,348 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c483d59d8c8a4d98b48549ad1b623e1c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682175227397 2023-04-22 14:53:50,348 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 92d36ac484ee439d96166340862e29f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682175228314 2023-04-22 14:53:50,349 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c540700feafd418c8876d078ad2c2de1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228496 2023-04-22 14:53:50,349 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:50,362 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#32 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:50,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:50,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:50,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:50,364 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:50,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:50,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:50,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:50,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,396 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,396 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,396 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:50,396 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:50,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:50,417 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/f6ef852b218c46ef8154a5792049f8e5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/f6ef852b218c46ef8154a5792049f8e5 2023-04-22 14:53:50,430 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into f6ef852b218c46ef8154a5792049f8e5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:50,430 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:50,430 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=13, startTime=1682175230230; duration=0sec 2023-04-22 14:53:50,431 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:50,431 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/5a4c7dcdf87e47bbb5dfc163b911196c 2023-04-22 14:53:50,450 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/961d3be9bf944fffa19503d5c5b6b051 2023-04-22 14:53:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:50,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:50,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175230721, max timestamp=1682175230731Num uniques 859; 2023-04-22 14:53:50,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:50,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175230721, max timestamp=1682175230731Num uniques 859; 2023-04-22 14:53:50,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175230721, max timestamp=1682175230731Num uniques 859; 2023-04-22 14:53:50,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:50,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:50,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175290785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175290785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175230763, max timestamp=1682175230790Num uniques 1018; 2023-04-22 14:53:50,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175230763, max timestamp=1682175230790Num uniques 1018; 2023-04-22 14:53:50,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:50,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:50,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175230763, max timestamp=1682175230790Num uniques 1018; 2023-04-22 14:53:50,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:53:50,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175290834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175290837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175290840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,875 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/27e967a0f86842c9a1ea7ffc0b6b7927 2023-04-22 14:53:50,883 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/5a4c7dcdf87e47bbb5dfc163b911196c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c 2023-04-22 14:53:50,891 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c, entries=150, sequenceid=289, filesize=11.9 K 2023-04-22 14:53:50,893 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/961d3be9bf944fffa19503d5c5b6b051 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051 2023-04-22 14:53:50,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175290896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175290895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,920 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051, entries=150, sequenceid=289, filesize=11.9 K 2023-04-22 14:53:50,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/27e967a0f86842c9a1ea7ffc0b6b7927 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927 2023-04-22 14:53:50,937 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927, entries=150, sequenceid=289, filesize=11.9 K 2023-04-22 14:53:50,938 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 58845d2ad95f1e691d0bdc0dfabebf52 in 542ms, sequenceid=289, compaction requested=false 2023-04-22 14:53:50,938 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:50,938 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:50,941 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:50,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8d6458[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:50,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:50,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=147.60 KB heapSize=420 KB 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:50,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:53:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:50,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,958 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:50,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:50,958 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60684 deadline: 1682175290961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:50,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175290963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175290963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:50,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:50,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:50,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:50,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:50,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,977 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:51,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175291065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175291069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175291073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,077 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:51,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:51,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:51,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:51,081 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:51,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:51,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:51,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,085 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,085 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:51,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:51,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:51,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:51,090 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,090 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:51,090 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:51,090 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:51,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:51,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc905cf[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:51,095 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:51,095 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,096 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:51,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:51,098 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:51,103 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:51,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:51,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:51,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:51,103 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:51,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175291102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,112 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:51,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,116 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:51,119 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:51,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175291121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,219 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:51,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:51,224 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:51,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:51,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:51,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:51,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:51,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:51,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:51,236 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,236 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:51,236 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:51,236 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:51,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e4372e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:51,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:51,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:51,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:51,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:51,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:51,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:51,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:51,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:51,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:51,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:51,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,268 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,268 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175291270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175291271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60642 deadline: 1682175291275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,368 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:51,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:51,373 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:51,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:51,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:51,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,375 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:51,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:51,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:51,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:51,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7acf63[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:51,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:51,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:51,384 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:51,387 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:51,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:51,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:51,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:51,388 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:51,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/50680bdeb61d4da7a3e04cfd57990762 2023-04-22 14:53:51,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:51,397 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:51,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:51,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:51,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:51,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175291408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175291426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,498 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:51,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:51,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:51,503 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:51,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:51,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:51,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:51,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:51,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:51,579 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,579 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:51,579 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:51,579 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:51,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175291579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175291580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:51,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175291581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,607 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,608 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,608 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5933c370[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:51,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:51,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,716 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:51,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:51,792 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:51,797 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:51,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:51,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:51,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:51,797 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:51,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8a6a6fe5e3764e75ad990b1b86727d2e 2023-04-22 14:53:51,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6e1e12018fc54fcda66ab39b7c77ebe3 2023-04-22 14:53:51,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/50680bdeb61d4da7a3e04cfd57990762 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762 2023-04-22 14:53:51,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:51,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,862 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:51,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762, entries=200, sequenceid=345, filesize=14.3 K 2023-04-22 14:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8a6a6fe5e3764e75ad990b1b86727d2e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e 2023-04-22 14:53:51,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e, entries=150, sequenceid=345, filesize=11.9 K 2023-04-22 14:53:51,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6e1e12018fc54fcda66ab39b7c77ebe3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3 2023-04-22 14:53:51,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3, entries=150, sequenceid=345, filesize=11.9 K 2023-04-22 14:53:51,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.95 KB/430032, currentSize=40.25 KB/41220 for 58845d2ad95f1e691d0bdc0dfabebf52 in 941ms, sequenceid=345, compaction requested=true 2023-04-22 14:53:51,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:51,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:51,888 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:51,888 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:51,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:51,889 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:51,894 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:51,894 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:51,894 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,894 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/9ed1ddaa25c943ed887d47f3ef0cdf89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=38.4 K 2023-04-22 14:53:51,895 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:51,895 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:51,895 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9ed1ddaa25c943ed887d47f3ef0cdf89, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228496 2023-04-22 14:53:51,896 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:51,896 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0bdd4b943adf42f6acb35fb75c96903d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=36.0 K 2023-04-22 14:53:51,897 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5a4c7dcdf87e47bbb5dfc163b911196c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682175229576 2023-04-22 14:53:51,897 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0bdd4b943adf42f6acb35fb75c96903d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228496 2023-04-22 14:53:51,898 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 50680bdeb61d4da7a3e04cfd57990762, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230817 2023-04-22 14:53:51,898 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 961d3be9bf944fffa19503d5c5b6b051, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682175229576 2023-04-22 14:53:51,900 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8a6a6fe5e3764e75ad990b1b86727d2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230824 2023-04-22 14:53:51,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:51,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:51,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:51,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:51,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:51,909 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:51,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:51,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:51,909 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:51,910 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:51,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:51,917 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#40 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:51,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:51,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:53:51,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:51,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:51,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:51,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:51,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:51,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:51,922 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#39 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:51,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175231957, max timestamp=1682175231969Num uniques 859; 2023-04-22 14:53:51,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175231957, max timestamp=1682175231969Num uniques 859; 2023-04-22 14:53:51,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:51,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:51,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175231957, max timestamp=1682175231969Num uniques 859; 2023-04-22 14:53:51,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:52,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175292004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,010 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:52,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:52,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175292007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:52,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:52,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:52,015 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:52,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:52,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:52,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,046 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:52,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:52,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:52,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:52,049 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,049 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:52,049 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:52,049 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:52,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175292087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175292089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175292090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed342e4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:52,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:52,095 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175292109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175292112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,146 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:52,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:52,192 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:52,195 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:52,195 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:52,195 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:52,195 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:52,195 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:52,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175292314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175292321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:52,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,343 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:52,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:52,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:52,343 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:52,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:52,343 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:52,343 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:52,343 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,344 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/0d323a1fa51a41b3be518e39756c0391 2023-04-22 14:53:52,363 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/440f762377f64d159eac05009fdb32ab as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/440f762377f64d159eac05009fdb32ab 2023-04-22 14:53:52,370 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/707af0914e6b4831b6cfb06447b40327 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/707af0914e6b4831b6cfb06447b40327 2023-04-22 14:53:52,384 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into 707af0914e6b4831b6cfb06447b40327(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:52,384 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:52,384 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=13, startTime=1682175231888; duration=0sec 2023-04-22 14:53:52,384 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:52,384 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:53:52,385 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into 440f762377f64d159eac05009fdb32ab(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:52,385 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:52,385 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=13, startTime=1682175231888; duration=0sec 2023-04-22 14:53:52,386 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:52,386 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:53:52,386 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:52,386 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,386 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/f6ef852b218c46ef8154a5792049f8e5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=36.0 K 2023-04-22 14:53:52,387 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f6ef852b218c46ef8154a5792049f8e5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682175228496 2023-04-22 14:53:52,388 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 27e967a0f86842c9a1ea7ffc0b6b7927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682175229576 2023-04-22 14:53:52,388 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6e1e12018fc54fcda66ab39b7c77ebe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230824 2023-04-22 14:53:52,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/dc04b5ca436644b2be42c1c0e917aa09 2023-04-22 14:53:52,406 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:52,443 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:52,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:52,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:52,448 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:52,448 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,448 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:52,448 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:52,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,477 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:52,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:52,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:52,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:52,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:52,479 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,479 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:52,479 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:52,479 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:52,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,531 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,531 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ada4357[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:52,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:52,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:52,585 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,585 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:52,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175292618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,630 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:52,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175292630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:52,634 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:52,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:52,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:52,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:52,634 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:52,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:52,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,677 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:52,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,679 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:52,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:52,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:52,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:52,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:52,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:52,738 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:52,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,739 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:52,739 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:52,739 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,739 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,739 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,739 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:52,739 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:52,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/ef0f040fb4a349a5bf854e6fb68f82ed 2023-04-22 14:53:52,831 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/09260903888646e39e74b24bd26ad880 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/09260903888646e39e74b24bd26ad880 2023-04-22 14:53:52,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/0d323a1fa51a41b3be518e39756c0391 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391 2023-04-22 14:53:52,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:52,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:52,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:52,844 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into 09260903888646e39e74b24bd26ad880(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:52,844 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:52,844 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=13, startTime=1682175231889; duration=0sec 2023-04-22 14:53:52,844 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:52,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:52,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:52,845 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:52,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:52,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:52,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391, entries=200, sequenceid=355, filesize=14.3 K 2023-04-22 14:53:52,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/dc04b5ca436644b2be42c1c0e917aa09 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09 2023-04-22 14:53:52,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09, entries=150, sequenceid=355, filesize=11.9 K 2023-04-22 14:53:52,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/ef0f040fb4a349a5bf854e6fb68f82ed as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed 2023-04-22 14:53:52,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed, entries=150, sequenceid=355, filesize=11.9 K 2023-04-22 14:53:52,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 58845d2ad95f1e691d0bdc0dfabebf52 in 948ms, sequenceid=355, compaction requested=false 2023-04-22 14:53:52,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:52,877 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:52,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:52,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:52,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:52,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:52,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:52,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:52,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:52,879 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-22 14:53:52,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:52,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:52,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:52,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:52,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:52,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:53,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:53,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:53,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175293104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175293105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175293106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60638 deadline: 1682175293123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60666 deadline: 1682175293135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175293207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175293210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175293211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,298 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/4835b2dea02a4493bed0f4829add2bec 2023-04-22 14:53:53,319 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/20ab8bdc41834eac9566aaa7b63ff295 2023-04-22 14:53:53,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175293411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175293416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175293417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175293716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175293720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:53,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175293728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,734 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6e66a44296a349868a5b2d2bea9afcab 2023-04-22 14:53:53,743 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/4835b2dea02a4493bed0f4829add2bec as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec 2023-04-22 14:53:53,752 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec, entries=150, sequenceid=394, filesize=11.9 K 2023-04-22 14:53:53,754 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/20ab8bdc41834eac9566aaa7b63ff295 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295 2023-04-22 14:53:53,766 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295, entries=150, sequenceid=394, filesize=11.9 K 2023-04-22 14:53:53,768 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/6e66a44296a349868a5b2d2bea9afcab as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab 2023-04-22 14:53:53,776 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab, entries=150, sequenceid=394, filesize=11.9 K 2023-04-22 14:53:53,778 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 58845d2ad95f1e691d0bdc0dfabebf52 in 898ms, sequenceid=394, compaction requested=true 2023-04-22 14:53:53,778 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:53,778 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:53,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:53,781 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633284b8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:53,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:53,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:53,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:53,799 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:53,809 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:53,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:53,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:53,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:53,810 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:53,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:53,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:53,814 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:53,815 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,818 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:53,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:53,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:53,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:53,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:53,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:53,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:53,819 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:53,819 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:53,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:53,919 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:53,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:53,924 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:53,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:53,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:53,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:53,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:53,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:53,926 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:53,926 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:53,927 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:53,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:53,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:53,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:53,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:53,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:53,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:53,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:53,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:53,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:53,932 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:53:53,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:53,933 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:53,933 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:53,933 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:53,933 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:53,933 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:53,985 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/b235918ae5eb4ee394b394e9187439d7 2023-04-22 14:53:54,000 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/1dcad337eba5481fb21552b90afe8eda 2023-04-22 14:53:54,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:54,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:54,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175234146, max timestamp=1682175234175Num uniques 859; 2023-04-22 14:53:54,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175234146, max timestamp=1682175234175Num uniques 859; 2023-04-22 14:53:54,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:54,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175234146, max timestamp=1682175234175Num uniques 859; 2023-04-22 14:53:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:54,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175294204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175294207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175294222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175294227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175294232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175294311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175294313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,415 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/214fe8a85e214106ac4e665f71788a34 2023-04-22 14:53:54,425 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/b235918ae5eb4ee394b394e9187439d7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7 2023-04-22 14:53:54,433 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7, entries=150, sequenceid=404, filesize=11.9 K 2023-04-22 14:53:54,435 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/1dcad337eba5481fb21552b90afe8eda as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda 2023-04-22 14:53:54,448 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda, entries=150, sequenceid=404, filesize=11.9 K 2023-04-22 14:53:54,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/214fe8a85e214106ac4e665f71788a34 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34 2023-04-22 14:53:54,460 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34, entries=150, sequenceid=404, filesize=11.9 K 2023-04-22 14:53:54,462 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 58845d2ad95f1e691d0bdc0dfabebf52 in 530ms, sequenceid=404, compaction requested=true 2023-04-22 14:53:54,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:54,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:54,467 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245de3ed[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:54,469 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:54,469 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,470 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:54,471 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,471 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:54,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:54,476 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:54,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:54,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:54,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:54,476 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:54,479 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,479 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,479 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,482 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,490 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,490 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:54,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:54,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,491 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,491 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:54,491 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:54,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:54,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-22 14:53:54,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:54,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:54,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:54,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:54,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:54,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:54,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175294528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175294533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,591 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:54,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:54,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:54,597 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:54,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:54,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:54,598 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,598 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:54,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:54,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:54,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:54,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:54,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:54,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:54,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,613 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60dd41e0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:54,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:54,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,618 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:54,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:54,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:54,623 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:54,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:54,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:54,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:54,623 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:54,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:54,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:54,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175294634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175294635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,732 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:54,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:54,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:54,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:54,747 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:54,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:54,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:54,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:54,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:54,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:54,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:54,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:54,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:54,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38457657[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:54,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:54,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:54,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:54,767 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:54,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,767 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,767 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:54,774 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:54,774 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:54,774 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:54,774 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:54,774 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:54,779 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,779 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,779 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,794 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,794 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,794 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:54,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:54,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:54,795 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:54,794 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,794 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,795 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,795 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,796 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,796 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,796 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:54,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175294840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175294841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,895 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:54,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:54,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:54,904 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:54,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:54,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:54,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:54,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:54,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:54,914 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,914 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa522a3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:54,916 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:54,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,923 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:54,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:54,924 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:54,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:54,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:54,930 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:54,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:54,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:54,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:54,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:54,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:54,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:54,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,942 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/205a2acde41344caa28a1da37f54ef9c 2023-04-22 14:53:54,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:54,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:54,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:54,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:54,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:54,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:54,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:54,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:54,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:54,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:54,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:55,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:55,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:55,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:55,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:55,055 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:55,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:55,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:55,056 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,056 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,058 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,058 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,059 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:55,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:55,072 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,072 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:55,073 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:55,073 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,074 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:55,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cf231[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:55,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:55,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,081 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:55,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:55,086 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:55,092 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:55,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:55,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:55,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:55,093 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:55,095 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,096 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,096 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,099 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,103 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:55,103 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:55,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175295147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175295147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:55,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:55,218 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:55,218 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,218 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:55,218 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:55,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,220 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,221 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:55,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:55,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:55,228 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,228 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:55,228 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:55,228 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:55,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:55,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175295231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175295232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175295235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e746115[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:55,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:55,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:55,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:55,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:55,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:55,251 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:55,251 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:55,251 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:55,251 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:55,251 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:55,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:55,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,260 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:55,260 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:55,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,361 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:55,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:55,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:55,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:55,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:55,368 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:55,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,370 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:55,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:55,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:55,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:55,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:55,378 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,378 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:55,378 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:55,378 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:55,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/33cf074b07cd4d7e84c9420bd5fe7928 2023-04-22 14:53:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:55,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d904942[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:55,384 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:55,384 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:55,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:55,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:55,404 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:55,404 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:55,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:55,408 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:55,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:55,416 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,416 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:55,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:55,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:55,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:55,417 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:55,417 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:55,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/79f915e950bf483cbe2dfb9f09a32a54 2023-04-22 14:53:55,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/205a2acde41344caa28a1da37f54ef9c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c 2023-04-22 14:53:55,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c, entries=200, sequenceid=441, filesize=14.3 K 2023-04-22 14:53:55,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/33cf074b07cd4d7e84c9420bd5fe7928 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928 2023-04-22 14:53:55,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928, entries=150, sequenceid=441, filesize=11.9 K 2023-04-22 14:53:55,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/79f915e950bf483cbe2dfb9f09a32a54 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54 2023-04-22 14:53:55,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54, entries=150, sequenceid=441, filesize=11.9 K 2023-04-22 14:53:55,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 58845d2ad95f1e691d0bdc0dfabebf52 in 963ms, sequenceid=441, compaction requested=true 2023-04-22 14:53:55,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:55,482 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:53:55,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:55,483 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:53:55,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:55,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:55,485 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:53:55,485 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66254 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:53:55,485 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:55,485 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:55,486 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,486 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,486 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/440f762377f64d159eac05009fdb32ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=59.9 K 2023-04-22 14:53:55,486 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/707af0914e6b4831b6cfb06447b40327, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=64.7 K 2023-04-22 14:53:55,486 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 440f762377f64d159eac05009fdb32ab, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230824 2023-04-22 14:53:55,486 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 707af0914e6b4831b6cfb06447b40327, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230824 2023-04-22 14:53:55,503 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0d323a1fa51a41b3be518e39756c0391, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682175230953 2023-04-22 14:53:55,504 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 4835b2dea02a4493bed0f4829add2bec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682175231984 2023-04-22 14:53:55,505 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting b235918ae5eb4ee394b394e9187439d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682175233102 2023-04-22 14:53:55,512 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting dc04b5ca436644b2be42c1c0e917aa09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682175230953 2023-04-22 14:53:55,513 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 205a2acde41344caa28a1da37f54ef9c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:55,514 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 20ab8bdc41834eac9566aaa7b63ff295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682175231984 2023-04-22 14:53:55,515 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1dcad337eba5481fb21552b90afe8eda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682175233102 2023-04-22 14:53:55,515 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 33cf074b07cd4d7e84c9420bd5fe7928, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:55,517 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:55,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:55,528 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:55,529 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,529 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:55,529 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:55,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,532 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,532 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:55,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:55,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:55,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:55,538 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:55,550 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:55,560 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,560 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:55,560 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:55,585 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/499ce98e31d042319b816dba9b52e17a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/499ce98e31d042319b816dba9b52e17a 2023-04-22 14:53:55,596 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/15531ed47baa459db74ae6b688c66dc0 2023-04-22 14:53:55,601 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/9fde884d7dd84d5a8938512a3b0a9f9b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9fde884d7dd84d5a8938512a3b0a9f9b 2023-04-22 14:53:55,603 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into 499ce98e31d042319b816dba9b52e17a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:55,603 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:55,603 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=11, startTime=1682175235482; duration=0sec 2023-04-22 14:53:55,603 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:55,604 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:53:55,608 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:53:55,608 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:55,608 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:55,608 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/09260903888646e39e74b24bd26ad880, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=59.9 K 2023-04-22 14:53:55,609 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 09260903888646e39e74b24bd26ad880, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682175230824 2023-04-22 14:53:55,609 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ef0f040fb4a349a5bf854e6fb68f82ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1682175230953 2023-04-22 14:53:55,610 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6e66a44296a349868a5b2d2bea9afcab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1682175231984 2023-04-22 14:53:55,610 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 214fe8a85e214106ac4e665f71788a34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682175233102 2023-04-22 14:53:55,611 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 79f915e950bf483cbe2dfb9f09a32a54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:55,624 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into 9fde884d7dd84d5a8938512a3b0a9f9b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:55,637 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:55,637 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=11, startTime=1682175235483; duration=0sec 2023-04-22 14:53:55,637 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:55,639 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:55,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:55,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:55,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175235696, max timestamp=1682175235713Num uniques 859; 2023-04-22 14:53:55,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:53:55,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175235696, max timestamp=1682175235713Num uniques 859; 2023-04-22 14:53:55,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:53:55,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175235696, max timestamp=1682175235713Num uniques 859; 2023-04-22 14:53:55,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:53:55,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175295747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175295747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175295851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:55,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:55,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175295851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,049 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/3f17be24a6194de3829af966c93cff32 2023-04-22 14:53:56,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175296054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:56,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175296056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,065 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/d623f15705ca4984af29165150c14226 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/d623f15705ca4984af29165150c14226 2023-04-22 14:53:56,072 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into d623f15705ca4984af29165150c14226(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:56,072 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:56,072 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=11, startTime=1682175235483; duration=0sec 2023-04-22 14:53:56,072 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175296357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175296360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,467 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/4f54d36d0c234492af47a2560810331b 2023-04-22 14:53:56,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/15531ed47baa459db74ae6b688c66dc0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0 2023-04-22 14:53:56,501 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0, entries=150, sequenceid=450, filesize=11.9 K 2023-04-22 14:53:56,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/3f17be24a6194de3829af966c93cff32 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32 2023-04-22 14:53:56,514 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32, entries=150, sequenceid=450, filesize=11.9 K 2023-04-22 14:53:56,516 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/4f54d36d0c234492af47a2560810331b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b 2023-04-22 14:53:56,529 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b, entries=150, sequenceid=450, filesize=11.9 K 2023-04-22 14:53:56,534 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 58845d2ad95f1e691d0bdc0dfabebf52 in 974ms, sequenceid=450, compaction requested=false 2023-04-22 14:53:56,534 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:56,534 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:56,549 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc6d069[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:56,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:56,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,553 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:56,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:56,565 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:56,569 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:56,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:56,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:56,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:56,569 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:56,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:56,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,583 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:56,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:56,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:56,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:56,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:56,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:56,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:56,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:56,684 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:56,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:56,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:56,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:56,692 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:56,692 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:56,692 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:56,692 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:56,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,694 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:56,694 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:56,695 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,695 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:56,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:56,702 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:56,702 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:56,702 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:56,702 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:56,702 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-22 14:53:56,702 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:56,702 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:56,702 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:56,703 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:56,703 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:56,703 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:56,720 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/54dae5872d5e41d396743f4753c9a95b 2023-04-22 14:53:56,760 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/406ca7efb48d40bfbbd56ddf29fd3127 2023-04-22 14:53:56,825 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/306f0c6046574c00a3cc095347a7165c 2023-04-22 14:53:56,835 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/54dae5872d5e41d396743f4753c9a95b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b 2023-04-22 14:53:56,847 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b, entries=150, sequenceid=489, filesize=11.9 K 2023-04-22 14:53:56,849 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/406ca7efb48d40bfbbd56ddf29fd3127 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127 2023-04-22 14:53:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:56,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:56,877 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127, entries=150, sequenceid=489, filesize=11.9 K 2023-04-22 14:53:56,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/306f0c6046574c00a3cc095347a7165c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c 2023-04-22 14:53:56,887 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c, entries=150, sequenceid=489, filesize=11.9 K 2023-04-22 14:53:56,888 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=13.42 KB/13740 for 58845d2ad95f1e691d0bdc0dfabebf52 in 186ms, sequenceid=489, compaction requested=true 2023-04-22 14:53:56,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:56,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:56,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:56,902 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1818185a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:56,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:56,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:56,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:56,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:56,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:56,910 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:56,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:56,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:56,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:56,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:56,914 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:56,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:56,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:56,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:56,914 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:56,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:56,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:56,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,920 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:56,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:56,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:56,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:56,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:56,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:56,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,926 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:56,927 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:56,927 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:56,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:56,927 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:56,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:56,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/a978aa2b89504d0db6eede5313410adb 2023-04-22 14:53:56,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175236959, max timestamp=1682175236970Num uniques 859; 2023-04-22 14:53:56,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:56,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175236959, max timestamp=1682175236970Num uniques 859; 2023-04-22 14:53:56,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:56,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175236959, max timestamp=1682175236970Num uniques 859; 2023-04-22 14:53:56,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:56,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/79f82d07d09f4a6fac1f71a481ac8361 2023-04-22 14:53:57,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/1f81dc4ad5564dd5bf6393ce903201a9 2023-04-22 14:53:57,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/a978aa2b89504d0db6eede5313410adb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb 2023-04-22 14:53:57,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175297016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175297021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,027 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb, entries=300, sequenceid=499, filesize=19.0 K 2023-04-22 14:53:57,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/79f82d07d09f4a6fac1f71a481ac8361 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361 2023-04-22 14:53:57,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,033 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,035 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361, entries=150, sequenceid=499, filesize=11.9 K 2023-04-22 14:53:57,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/1f81dc4ad5564dd5bf6393ce903201a9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9 2023-04-22 14:53:57,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,040 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,040 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,041 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,041 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:57,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:57,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:57,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:57,042 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,042 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:57,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9, entries=150, sequenceid=499, filesize=11.9 K 2023-04-22 14:53:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 58845d2ad95f1e691d0bdc0dfabebf52 in 135ms, sequenceid=499, compaction requested=true 2023-04-22 14:53:57,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:57,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:57,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:57,045 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:57,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3549ae85[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:57,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:57,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,047 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:57,047 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56678 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:57,047 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:53:57,047 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:53:57,047 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,047 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,047 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9fde884d7dd84d5a8938512a3b0a9f9b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=48.2 K 2023-04-22 14:53:57,047 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/499ce98e31d042319b816dba9b52e17a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=55.3 K 2023-04-22 14:53:57,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,048 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9fde884d7dd84d5a8938512a3b0a9f9b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:57,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:57,048 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 499ce98e31d042319b816dba9b52e17a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:57,048 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3f17be24a6194de3829af966c93cff32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682175234522 2023-04-22 14:53:57,048 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 15531ed47baa459db74ae6b688c66dc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682175234522 2023-04-22 14:53:57,049 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 406ca7efb48d40bfbbd56ddf29fd3127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682175235738 2023-04-22 14:53:57,049 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 54dae5872d5e41d396743f4753c9a95b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682175235738 2023-04-22 14:53:57,049 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 79f82d07d09f4a6fac1f71a481ac8361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236902 2023-04-22 14:53:57,049 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a978aa2b89504d0db6eede5313410adb, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236869 2023-04-22 14:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,050 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:57,055 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:57,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:57,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:57,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:57,055 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:57,059 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,059 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,059 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,066 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:57,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,066 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:57,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,069 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:57,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:57,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:57,069 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:57,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:57,103 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/401a29bac981453e88c36c9c121badbe as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/401a29bac981453e88c36c9c121badbe 2023-04-22 14:53:57,107 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8b5f8ca5fae443c79744097451f1a07c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8b5f8ca5fae443c79744097451f1a07c 2023-04-22 14:53:57,113 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into 401a29bac981453e88c36c9c121badbe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:57,113 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:57,113 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=12, startTime=1682175237045; duration=0sec 2023-04-22 14:53:57,114 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:57,114 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:57,116 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:53:57,116 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:53:57,116 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,116 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into 8b5f8ca5fae443c79744097451f1a07c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:57,116 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:57,116 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=12, startTime=1682175237045; duration=0sec 2023-04-22 14:53:57,117 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:57,117 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/d623f15705ca4984af29165150c14226, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=48.2 K 2023-04-22 14:53:57,118 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d623f15705ca4984af29165150c14226, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1682175234196 2023-04-22 14:53:57,119 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 4f54d36d0c234492af47a2560810331b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682175234522 2023-04-22 14:53:57,120 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 306f0c6046574c00a3cc095347a7165c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682175235738 2023-04-22 14:53:57,121 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1f81dc4ad5564dd5bf6393ce903201a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236902 2023-04-22 14:53:57,133 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:57,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175237001, max timestamp=1682175237132Num uniques 1018; 2023-04-22 14:53:57,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:53:57,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:57,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-22 14:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175237001, max timestamp=1682175237132Num uniques 1018; 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175237001, max timestamp=1682175237132Num uniques 1018; 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:57,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/e82fe54b78a3457584a16f2413a2caf3 2023-04-22 14:53:57,164 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/a1a9afdfbe3540bd8ce2ac8050cbba31 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/a1a9afdfbe3540bd8ce2ac8050cbba31 2023-04-22 14:53:57,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,185 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,185 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,185 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,185 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/0f46efbe9f9a42b6b6b3cbed67b07be7 2023-04-22 14:53:57,186 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,186 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,188 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,188 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,197 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,197 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,197 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,197 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:57,199 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8c2b37[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:57,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:57,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into a1a9afdfbe3540bd8ce2ac8050cbba31(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:57,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:57,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=12, startTime=1682175237045; duration=0sec 2023-04-22 14:53:57,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:57,204 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:57,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:57,206 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:57,215 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:57,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:57,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:57,216 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:57,216 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:57,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175237169, max timestamp=1682175237213Num uniques 859; 2023-04-22 14:53:57,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:57,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175237169, max timestamp=1682175237213Num uniques 859; 2023-04-22 14:53:57,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:57,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175237169, max timestamp=1682175237213Num uniques 859; 2023-04-22 14:53:57,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:53:57,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:57,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:57,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:57,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,229 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:57,229 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682175237248, max timestamp=1682175237256Num uniques 1068; 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682175237248, max timestamp=1682175237256Num uniques 1068; 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682175237248, max timestamp=1682175237256Num uniques 1068; 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-22 14:53:57,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-22 14:53:57,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175297289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175297289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175297291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175297292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175297292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,329 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,334 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,335 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,335 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,340 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,340 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,340 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,340 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:57,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6784f397[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:57,344 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:57,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:57,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:57,348 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:57,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:57,351 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:57,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:57,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:57,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:57,351 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:57,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,354 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:57,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:57,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:57,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:57,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175297394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175297394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175297394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175297396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175297396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,464 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,464 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,464 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,464 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,467 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:57,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f86edf5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:57,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:57,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:57,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:57,477 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:57,479 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:57,483 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:57,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:57,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:57,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:57,483 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:57,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,486 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,492 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:57,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:57,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:57,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175297599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175297600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175297600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175297600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175297600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,608 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,610 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,623 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,623 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,623 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,623 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:57,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:57,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c34106[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:57,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:57,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:57,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:57,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7a9da98be3284b8d80fade441b6a8f11 2023-04-22 14:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:57,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:57,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:57,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:57,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:57,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:57,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:57,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:57,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/e82fe54b78a3457584a16f2413a2caf3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3 2023-04-22 14:53:57,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3, entries=250, sequenceid=540, filesize=16.7 K 2023-04-22 14:53:57,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/0f46efbe9f9a42b6b6b3cbed67b07be7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:57,654 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:57,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:57,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:57,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:57,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:57,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:57,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:57,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7, entries=150, sequenceid=540, filesize=11.9 K 2023-04-22 14:53:57,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7a9da98be3284b8d80fade441b6a8f11 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11 2023-04-22 14:53:57,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11, entries=150, sequenceid=540, filesize=11.9 K 2023-04-22 14:53:57,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=154.31 KB/158010 for 58845d2ad95f1e691d0bdc0dfabebf52 in 538ms, sequenceid=540, compaction requested=false 2023-04-22 14:53:57,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:57,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:57,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:57,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:57,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:57,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:57,759 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:57,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:57,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:57,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:57,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:57,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:57,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:57,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:57,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:57,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:57,764 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:57,783 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/8c0cc45469e04cfd9b6a637ce145ff35 2023-04-22 14:53:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:57,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175297913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175297913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60700 deadline: 1682175297924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175297924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:57,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175297925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175298020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175298026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175298026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175298028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,197 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/6ad935ace09443ee9a4c6291d4dd62c0 2023-04-22 14:53:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175298224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175298228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175298231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175298231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175298421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175298526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175298533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175298534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175298536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,612 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/c3d2d220c71c4c578fc21a1cc03b1315 2023-04-22 14:53:58,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/8c0cc45469e04cfd9b6a637ce145ff35 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35 2023-04-22 14:53:58,630 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35, entries=150, sequenceid=595, filesize=11.9 K 2023-04-22 14:53:58,631 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/6ad935ace09443ee9a4c6291d4dd62c0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0 2023-04-22 14:53:58,642 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0, entries=150, sequenceid=595, filesize=11.9 K 2023-04-22 14:53:58,644 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/c3d2d220c71c4c578fc21a1cc03b1315 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315 2023-04-22 14:53:58,655 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315, entries=150, sequenceid=595, filesize=11.9 K 2023-04-22 14:53:58,657 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=40.25 KB/41220 for 58845d2ad95f1e691d0bdc0dfabebf52 in 893ms, sequenceid=595, compaction requested=true 2023-04-22 14:53:58,657 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:58,657 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:58,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:58,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,660 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,660 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d1f5a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:58,662 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:58,662 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,663 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:58,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:58,666 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:58,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:58,670 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:58,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:58,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:58,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:58,670 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:58,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,675 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,677 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:58,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:58,678 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:58,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:58,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:58,779 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:58,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:58,789 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:58,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:58,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:58,790 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,790 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:58,795 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,795 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:58,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:58,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:58,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:58,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:58,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:58,816 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:58,816 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:58,816 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:53:59,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:59,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175239100, max timestamp=1682175239104Num uniques 859; 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175239100, max timestamp=1682175239104Num uniques 859; 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175239100, max timestamp=1682175239104Num uniques 859; 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:53:59,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682175239147, max timestamp=1682175239151Num uniques 1018; 2023-04-22 14:53:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-22 14:53:59,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682175239147, max timestamp=1682175239151Num uniques 1018; 2023-04-22 14:53:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175299149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-22 14:53:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175239148, max timestamp=1682175239151Num uniques 1018; 2023-04-22 14:53:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-22 14:53:59,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175299177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175299180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175299181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,236 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/f9bbcccde1a849ac82a18444bdb4dbbe 2023-04-22 14:53:59,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175299255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175299282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175299285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175299285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175299429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175299458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175299488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175299489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175299490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,665 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/f843915f511e4da1a733782c83bc1162 2023-04-22 14:53:59,686 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/863e92aec9334369a542739bd25e6745 2023-04-22 14:53:59,694 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/f9bbcccde1a849ac82a18444bdb4dbbe as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe 2023-04-22 14:53:59,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe, entries=150, sequenceid=604, filesize=11.9 K 2023-04-22 14:53:59,701 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/f843915f511e4da1a733782c83bc1162 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162 2023-04-22 14:53:59,708 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162, entries=150, sequenceid=604, filesize=11.9 K 2023-04-22 14:53:59,709 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/863e92aec9334369a542739bd25e6745 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745 2023-04-22 14:53:59,717 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745, entries=150, sequenceid=604, filesize=11.9 K 2023-04-22 14:53:59,718 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=142.33 KB/145742 for 58845d2ad95f1e691d0bdc0dfabebf52 in 902ms, sequenceid=604, compaction requested=true 2023-04-22 14:53:59,718 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:53:59,718 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:59,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:59,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c055d0d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,724 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:59,724 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,725 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:59,726 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,726 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,733 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:59,733 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,733 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,733 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:59,733 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,735 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,735 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,736 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,737 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:59,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,744 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,744 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:53:59,770 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=149.04 KB heapSize=423.25 KB 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:53:59,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175239175, max timestamp=1682175239766Num uniques 1127; 2023-04-22 14:53:59,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175239175, max timestamp=1682175239766Num uniques 1009; 2023-04-22 14:53:59,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-22 14:53:59,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:53:59,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175299802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175299803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175299803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175299804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,844 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:59,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:59,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:59,848 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:59,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:59,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:59,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,850 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:59,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:59,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:59,851 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:59,851 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,851 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:59,852 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:59,852 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:59,852 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:59,852 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:59,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4464e83d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:59,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:59,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,863 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:59,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:59,864 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:53:59,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,873 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:53:59,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175299909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175299909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175299910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:53:59,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175299910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,974 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:59,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:53:59,977 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:53:59,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:53:59,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:53:59,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,984 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,984 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,984 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,984 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:53:59,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:53:59,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:53:59,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:59,986 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:59,986 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:53:59,986 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:53:59,986 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:53:59,988 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:53:59,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19725ac6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:53:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:53:59,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:53:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:53:59,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:53:59,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,995 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:53:59,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:53:59,995 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:53:59,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,002 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,106 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,111 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175300113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175300113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,117 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175300116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175300116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db1217d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,129 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,129 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,131 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,136 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/3f44224854f142d8a8cf7f995f75113e 2023-04-22 14:54:00,236 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,239 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,242 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,246 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbae6f3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,252 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,253 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,253 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,255 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,255 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,262 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,366 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,368 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcc0b9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,378 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,383 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,383 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,385 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,385 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,391 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175300415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175300416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175300420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175300422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,491 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,495 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,504 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7353010a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,508 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,508 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,511 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,512 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,515 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,519 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,519 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/9d96df179519497b9c3b5ce34a99c81e 2023-04-22 14:54:00,620 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,623 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d7cd4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,637 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,637 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,645 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,645 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,745 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,749 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,754 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bdc8faa[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,769 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,769 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,771 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,773 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:00,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:00,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:00,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:00,880 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:00,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:00,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:00,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,882 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,882 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:00,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:00,885 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,885 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:00,885 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:00,885 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:00,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:00,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:00,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:00,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:00,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:00,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55bb6760[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:00,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:00,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:00,893 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:00,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:00,894 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:00,898 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:00,898 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:00,898 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:00,898 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:00,898 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:00,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:00,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:00,902 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,903 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:00,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:00,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:00,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:00,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:00,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:00,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:00,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:00,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:00,908 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:00,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:00,908 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:00,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:00,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175300917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175300919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175300924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:00,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:00,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175300927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/e42f4c411fa64d51be994c2119a5c00d 2023-04-22 14:54:01,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:01,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:01,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:01,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:01,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:01,014 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:01,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:01,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:01,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/3f44224854f142d8a8cf7f995f75113e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e 2023-04-22 14:54:01,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,017 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,017 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:01,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:01,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:01,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:01,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:01,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:01,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:01,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:01,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e, entries=200, sequenceid=659, filesize=14.3 K 2023-04-22 14:54:01,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/9d96df179519497b9c3b5ce34a99c81e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e 2023-04-22 14:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0b678c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:01,027 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:01,027 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:01,030 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:01,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e, entries=150, sequenceid=659, filesize=11.9 K 2023-04-22 14:54:01,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/e42f4c411fa64d51be994c2119a5c00d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d 2023-04-22 14:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:01,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:01,037 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:01,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:01,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:01,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:01,037 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:01,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d, entries=150, sequenceid=659, filesize=11.9 K 2023-04-22 14:54:01,039 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,039 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:01,039 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~188.06 KB/192576, currentSize=114.05 KB/116790 for 58845d2ad95f1e691d0bdc0dfabebf52 in 1269ms, sequenceid=659, compaction requested=true 2023-04-22 14:54:01,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:01,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:01,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:01,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:01,040 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:01,040 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,041 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,042 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 69000 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:01,042 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,042 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,043 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/401a29bac981453e88c36c9c121badbe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=67.4 K 2023-04-22 14:54:01,043 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:01,043 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,043 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,043 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,043 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8b5f8ca5fae443c79744097451f1a07c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=60.2 K 2023-04-22 14:54:01,043 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 401a29bac981453e88c36c9c121badbe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236902 2023-04-22 14:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,044 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8b5f8ca5fae443c79744097451f1a07c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236902 2023-04-22 14:54:01,044 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e82fe54b78a3457584a16f2413a2caf3, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682175237001 2023-04-22 14:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,045 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8c0cc45469e04cfd9b6a637ce145ff35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1682175237285 2023-04-22 14:54:01,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0f46efbe9f9a42b6b6b3cbed67b07be7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682175237001 2023-04-22 14:54:01,045 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f9bbcccde1a849ac82a18444bdb4dbbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682175237910 2023-04-22 14:54:01,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6ad935ace09443ee9a4c6291d4dd62c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1682175237285 2023-04-22 14:54:01,046 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3f44224854f142d8a8cf7f995f75113e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,046 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f843915f511e4da1a733782c83bc1162, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682175237910 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,047 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9d96df179519497b9c3b5ce34a99c81e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,047 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:01,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:01,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:01,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:01,047 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:01,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:01,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:01,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:01,060 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:01,062 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:01,079 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/e636bf58707c4e9690ce5a8d0b87900d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e636bf58707c4e9690ce5a8d0b87900d 2023-04-22 14:54:01,086 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into e636bf58707c4e9690ce5a8d0b87900d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:01,086 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:01,086 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=11, startTime=1682175241040; duration=0sec 2023-04-22 14:54:01,086 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:01,086 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:01,089 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:01,089 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:54:01,089 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,089 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/a1a9afdfbe3540bd8ce2ac8050cbba31, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=60.2 K 2023-04-22 14:54:01,090 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a1a9afdfbe3540bd8ce2ac8050cbba31, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682175236902 2023-04-22 14:54:01,090 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7a9da98be3284b8d80fade441b6a8f11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682175237001 2023-04-22 14:54:01,090 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c3d2d220c71c4c578fc21a1cc03b1315, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1682175237285 2023-04-22 14:54:01,091 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 863e92aec9334369a542739bd25e6745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682175237910 2023-04-22 14:54:01,091 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e42f4c411fa64d51be994c2119a5c00d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:01,102 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:01,112 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/9060f90211c54d9a94d95648c252f3ef as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/9060f90211c54d9a94d95648c252f3ef 2023-04-22 14:54:01,118 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into 9060f90211c54d9a94d95648c252f3ef(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:01,118 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:01,119 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=11, startTime=1682175241040; duration=0sec 2023-04-22 14:54:01,119 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:01,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:01,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:01,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:01,152 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:01,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:01,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:01,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:01,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:01,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:01,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:01,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:01,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:01,156 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:01,157 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,165 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/c29e3d5711974ce59f68e3dd14fef5e3 2023-04-22 14:54:01,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:01,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:01,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175301468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,488 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/c87a1ac7d65e434fad80ba3d3f60ce1a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c87a1ac7d65e434fad80ba3d3f60ce1a 2023-04-22 14:54:01,498 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into c87a1ac7d65e434fad80ba3d3f60ce1a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:01,498 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:01,498 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=11, startTime=1682175241040; duration=0sec 2023-04-22 14:54:01,498 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:01,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175301570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,581 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/4744837cbebb408b955c840baa134a92 2023-04-22 14:54:01,594 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/08602eb04c1345d0815fcda008b09e57 2023-04-22 14:54:01,600 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/c29e3d5711974ce59f68e3dd14fef5e3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3 2023-04-22 14:54:01,604 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 14:54:01,606 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/4744837cbebb408b955c840baa134a92 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92 2023-04-22 14:54:01,611 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 14:54:01,612 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/08602eb04c1345d0815fcda008b09e57 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57 2023-04-22 14:54:01,616 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57, entries=150, sequenceid=681, filesize=11.9 K 2023-04-22 14:54:01,617 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 58845d2ad95f1e691d0bdc0dfabebf52 in 461ms, sequenceid=681, compaction requested=false 2023-04-22 14:54:01,617 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:01,617 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:01,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:01,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:01,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:01,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@529cd745[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:01,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,623 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:01,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:01,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:01,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:01,628 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:01,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:01,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:01,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:01,628 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:01,629 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,629 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,629 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:01,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:01,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:01,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:01,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:01,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:01,638 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,639 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:01,639 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:01,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:01,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:01,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:01,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:01,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:01,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:01,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:01,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:01,744 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:01,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,747 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:01,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:01,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:01,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:01,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:01,761 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:01,761 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:01,761 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:01,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:01,775 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/660e78be1077424fa54c5b3e069ab4de 2023-04-22 14:54:01,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175301816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175301918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175301920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175301930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175301933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:01,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:01,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175301943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175302123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,188 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8ef09fb5a1d14d84ab8e555a8e86d8fa 2023-04-22 14:54:02,205 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/e7bfb31dec7d4afabd6d00191601ea8c 2023-04-22 14:54:02,210 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/660e78be1077424fa54c5b3e069ab4de as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de 2023-04-22 14:54:02,215 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de, entries=150, sequenceid=694, filesize=11.9 K 2023-04-22 14:54:02,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/8ef09fb5a1d14d84ab8e555a8e86d8fa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa 2023-04-22 14:54:02,222 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa, entries=150, sequenceid=694, filesize=11.9 K 2023-04-22 14:54:02,223 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/e7bfb31dec7d4afabd6d00191601ea8c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c 2023-04-22 14:54:02,228 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c, entries=150, sequenceid=694, filesize=11.9 K 2023-04-22 14:54:02,232 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 58845d2ad95f1e691d0bdc0dfabebf52 in 471ms, sequenceid=694, compaction requested=true 2023-04-22 14:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:02,234 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:02,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:02,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d12a363[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:02,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:02,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:02,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:02,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:02,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:02,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:02,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:02,243 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:02,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:02,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:02,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:02,243 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:02,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:02,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:02,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:02,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:02,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:02,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:02,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:02,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:02,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:02,249 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:02,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:02,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:02,349 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:02,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:02,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:02,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:02,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:02,356 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:02,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:02,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:02,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:02,357 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,357 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:02,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,358 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:02,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:02,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:02,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:02,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:02,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:02,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:02,364 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:02,365 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:02,365 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:02,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:02,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:02,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:02,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175302459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175302564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175302765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:02,775 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/0a837c5a7a534e968bcd1db933bc4b07 2023-04-22 14:54:02,801 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/d50b56e6b24f49939fc69c993faddf9f 2023-04-22 14:54:03,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175303070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,217 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/0b785acbcc35406db3710418db57dcac 2023-04-22 14:54:03,223 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/0a837c5a7a534e968bcd1db933bc4b07 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07 2023-04-22 14:54:03,228 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07, entries=150, sequenceid=714, filesize=11.9 K 2023-04-22 14:54:03,229 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/d50b56e6b24f49939fc69c993faddf9f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f 2023-04-22 14:54:03,234 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f, entries=150, sequenceid=714, filesize=11.9 K 2023-04-22 14:54:03,235 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/0b785acbcc35406db3710418db57dcac as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac 2023-04-22 14:54:03,241 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac, entries=150, sequenceid=714, filesize=11.9 K 2023-04-22 14:54:03,242 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 58845d2ad95f1e691d0bdc0dfabebf52 in 877ms, sequenceid=714, compaction requested=true 2023-04-22 14:54:03,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:03,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:03,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:03,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731c4bc6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:03,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:03,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:03,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:03,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:03,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:03,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:03,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:03,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:03,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:03,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:03,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:03,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:03,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:03,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:03,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:03,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:03,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:03,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:03,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:03,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:03,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:03,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:03,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:03,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:03,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:03,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:03,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:03,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:03,380 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:03,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:03,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:03,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:03,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:03,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:03,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:03,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:03,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:03,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:03,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:03,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:03,396 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:03,396 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. started... 2023-04-22 14:54:03,396 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:03,397 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:03,408 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/56110c8a5a314243a0148efbfa7d9e6a 2023-04-22 14:54:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:03,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. as already flushing 2023-04-22 14:54:03,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175303639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175303745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,786 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:51853 2023-04-22 14:54:03,786 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:03,825 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:51853 2023-04-22 14:54:03,825 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:03,826 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:51853 2023-04-22 14:54:03,826 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:03,826 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:51853 2023-04-22 14:54:03,827 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:03,857 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/00066833dace496bb970ec0ce0da984a 2023-04-22 14:54:03,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60642 deadline: 1682175303935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60666 deadline: 1682175303935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,937 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:03,937 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:03,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60638 deadline: 1682175303942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,944 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:03,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175303949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:03,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60700 deadline: 1682175303959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:03,960 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:04,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:04,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60684 deadline: 1682175304251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,272 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/205b0fec999742c4bd8e75e57546488c 2023-04-22 14:54:04,277 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/56110c8a5a314243a0148efbfa7d9e6a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a 2023-04-22 14:54:04,282 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a, entries=150, sequenceid=726, filesize=11.9 K 2023-04-22 14:54:04,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/00066833dace496bb970ec0ce0da984a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a 2023-04-22 14:54:04,287 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a, entries=150, sequenceid=726, filesize=11.9 K 2023-04-22 14:54:04,288 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/205b0fec999742c4bd8e75e57546488c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c 2023-04-22 14:54:04,293 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c, entries=150, sequenceid=726, filesize=11.9 K 2023-04-22 14:54:04,294 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 58845d2ad95f1e691d0bdc0dfabebf52 in 898ms, sequenceid=726, compaction requested=true 2023-04-22 14:54:04,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:04,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:04,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:04,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,296 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:04,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:04,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ce652e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:04,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:04,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,299 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:04,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:04,302 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:04,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:04,305 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:04,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:04,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:04,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:04,305 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:04,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:04,308 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,309 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,318 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:04,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:04,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:04,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:04,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:04,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:04,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:04,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:04,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:04,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:04,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:04,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:04,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:04,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-22 14:54:04,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:04,758 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:51853 2023-04-22 14:54:04,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:04,758 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:04,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:04,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:04,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:04,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175243618, max timestamp=1682175244757Num uniques 859; 2023-04-22 14:54:04,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:04,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:05,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/3ace5f19360b44c891db70952f6b9fb8 2023-04-22 14:54:05,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/81067bbc660640d19af88e643882fdcb 2023-04-22 14:54:05,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/cf2105a71351410398acafcca078d942 2023-04-22 14:54:05,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/3ace5f19360b44c891db70952f6b9fb8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8 2023-04-22 14:54:05,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8, entries=150, sequenceid=747, filesize=11.9 K 2023-04-22 14:54:05,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/81067bbc660640d19af88e643882fdcb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb 2023-04-22 14:54:05,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb, entries=150, sequenceid=747, filesize=11.9 K 2023-04-22 14:54:05,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/cf2105a71351410398acafcca078d942 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942 2023-04-22 14:54:05,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942, entries=150, sequenceid=747, filesize=11.9 K 2023-04-22 14:54:05,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=0 B/0 for 58845d2ad95f1e691d0bdc0dfabebf52 in 852ms, sequenceid=747, compaction requested=true 2023-04-22 14:54:05,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:05,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:05,610 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:54:05,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:05,610 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:54:05,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:05,612 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:54:05,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:54:05,612 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/A is initiating minor compaction (all files) 2023-04-22 14:54:05,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/B is initiating minor compaction (all files) 2023-04-22 14:54:05,612 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/A in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:05,612 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/B in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:05,612 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c87a1ac7d65e434fad80ba3d3f60ce1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=72.3 K 2023-04-22 14:54:05,612 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e636bf58707c4e9690ce5a8d0b87900d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=72.3 K 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c87a1ac7d65e434fad80ba3d3f60ce1a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e636bf58707c4e9690ce5a8d0b87900d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c29e3d5711974ce59f68e3dd14fef5e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682175239801 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4744837cbebb408b955c840baa134a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682175239801 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 660e78be1077424fa54c5b3e069ab4de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682175241451 2023-04-22 14:54:05,613 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8ef09fb5a1d14d84ab8e555a8e86d8fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682175241451 2023-04-22 14:54:05,614 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0a837c5a7a534e968bcd1db933bc4b07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1682175241798 2023-04-22 14:54:05,614 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d50b56e6b24f49939fc69c993faddf9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1682175241798 2023-04-22 14:54:05,614 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 56110c8a5a314243a0148efbfa7d9e6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682175242432 2023-04-22 14:54:05,614 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 00066833dace496bb970ec0ce0da984a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682175242432 2023-04-22 14:54:05,615 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3ace5f19360b44c891db70952f6b9fb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1682175243632 2023-04-22 14:54:05,615 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 81067bbc660640d19af88e643882fdcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1682175243632 2023-04-22 14:54:05,627 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#A#compaction#99 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:05,627 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#B#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:06,037 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/d94917d518974534ac50d3436a7870ef as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/d94917d518974534ac50d3436a7870ef 2023-04-22 14:54:06,037 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/c1542f26d4e145daa6924f690fbb8e85 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/c1542f26d4e145daa6924f690fbb8e85 2023-04-22 14:54:06,042 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/B of 58845d2ad95f1e691d0bdc0dfabebf52 into c1542f26d4e145daa6924f690fbb8e85(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:06,042 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/A of 58845d2ad95f1e691d0bdc0dfabebf52 into d94917d518974534ac50d3436a7870ef(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:06,043 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:06,043 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:06,043 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/B, priority=10, startTime=1682175245610; duration=0sec 2023-04-22 14:54:06,043 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/A, priority=10, startTime=1682175245610; duration=0sec 2023-04-22 14:54:06,043 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:06,043 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:06,043 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:54:06,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:54:06,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 58845d2ad95f1e691d0bdc0dfabebf52/C is initiating minor compaction (all files) 2023-04-22 14:54:06,045 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 58845d2ad95f1e691d0bdc0dfabebf52/C in TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:06,045 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/9060f90211c54d9a94d95648c252f3ef, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp, totalSize=72.3 K 2023-04-22 14:54:06,045 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9060f90211c54d9a94d95648c252f3ef, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682175239175 2023-04-22 14:54:06,046 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 08602eb04c1345d0815fcda008b09e57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682175239801 2023-04-22 14:54:06,046 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e7bfb31dec7d4afabd6d00191601ea8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682175241451 2023-04-22 14:54:06,046 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0b785acbcc35406db3710418db57dcac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1682175241798 2023-04-22 14:54:06,046 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 205b0fec999742c4bd8e75e57546488c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682175242432 2023-04-22 14:54:06,047 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting cf2105a71351410398acafcca078d942, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1682175243632 2023-04-22 14:54:06,056 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 58845d2ad95f1e691d0bdc0dfabebf52#C#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:06,467 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/16b06ccd0ad246ec82ec48e11223b4b0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/16b06ccd0ad246ec82ec48e11223b4b0 2023-04-22 14:54:06,473 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 58845d2ad95f1e691d0bdc0dfabebf52/C of 58845d2ad95f1e691d0bdc0dfabebf52 into 16b06ccd0ad246ec82ec48e11223b4b0(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:06,473 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:06,473 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52., storeName=58845d2ad95f1e691d0bdc0dfabebf52/C, priority=10, startTime=1682175245610; duration=0sec 2023-04-22 14:54:06,474 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:07,947 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:51853 2023-04-22 14:54:07,948 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:51853 2023-04-22 14:54:07,948 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:07,948 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:07,963 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:51853 2023-04-22 14:54:07,963 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:07,987 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:51853 2023-04-22 14:54:07,987 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 152 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 3797 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 3806 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1697 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5087 rows 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1707 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5117 rows 2023-04-22 14:54:07,988 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:54:07,988 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:51853 2023-04-22 14:54:07,988 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:07,993 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:54:07,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:54:08,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:08,008 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175248008"}]},"ts":"1682175248008"} 2023-04-22 14:54:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:08,009 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:54:08,012 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:54:08,014 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, UNASSIGN}] 2023-04-22 14:54:08,015 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, UNASSIGN 2023-04-22 14:54:08,015 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=58845d2ad95f1e691d0bdc0dfabebf52, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:08,016 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175248015"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175248015"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175248015"}]},"ts":"1682175248015"} 2023-04-22 14:54:08,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:08,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:08,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 58845d2ad95f1e691d0bdc0dfabebf52, disabling compactions & flushes 2023-04-22 14:54:08,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. after waiting 0 ms 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:08,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 58845d2ad95f1e691d0bdc0dfabebf52 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=A 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=B 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:08,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 58845d2ad95f1e691d0bdc0dfabebf52, store=C 2023-04-22 14:54:08,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:08,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:08,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/274286b8863a4a1084e764aa0cda40dd 2023-04-22 14:54:08,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:08,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/68f9a4668ae948208bbc1b7d0982b778 2023-04-22 14:54:09,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:09,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7536a9ba1db7413ca1f94879a7d83a05 2023-04-22 14:54:09,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/A/274286b8863a4a1084e764aa0cda40dd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/274286b8863a4a1084e764aa0cda40dd 2023-04-22 14:54:09,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/274286b8863a4a1084e764aa0cda40dd, entries=100, sequenceid=757, filesize=9.5 K 2023-04-22 14:54:09,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/B/68f9a4668ae948208bbc1b7d0982b778 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/68f9a4668ae948208bbc1b7d0982b778 2023-04-22 14:54:09,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/68f9a4668ae948208bbc1b7d0982b778, entries=100, sequenceid=757, filesize=9.5 K 2023-04-22 14:54:09,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/.tmp/C/7536a9ba1db7413ca1f94879a7d83a05 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7536a9ba1db7413ca1f94879a7d83a05 2023-04-22 14:54:09,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7536a9ba1db7413ca1f94879a7d83a05, entries=100, sequenceid=757, filesize=9.5 K 2023-04-22 14:54:09,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 58845d2ad95f1e691d0bdc0dfabebf52 in 1253ms, sequenceid=757, compaction requested=false 2023-04-22 14:54:09,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b08146ea20b24bae96a6b62b1e8a3a35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/45e24fe447fd41f8a110c730c318cabe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/9ed1ddaa25c943ed887d47f3ef0cdf89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/707af0914e6b4831b6cfb06447b40327, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/499ce98e31d042319b816dba9b52e17a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/401a29bac981453e88c36c9c121badbe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c87a1ac7d65e434fad80ba3d3f60ce1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8] to archive 2023-04-22 14:54:09,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:09,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/bdc9177ebe524bde99fb1082488ac204 2023-04-22 14:54:09,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/158d7031a8fc472f84e6f425d813d0f5 2023-04-22 14:54:09,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/11b5d62e96674c1e95eecac589fbe39a 2023-04-22 14:54:09,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/ac4e52037e54418f9a5c870aacd45b84 2023-04-22 14:54:09,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b08146ea20b24bae96a6b62b1e8a3a35 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b08146ea20b24bae96a6b62b1e8a3a35 2023-04-22 14:54:09,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/2b4e9b4f9e7e43f6a1871d2ad05f26ee 2023-04-22 14:54:09,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/45e24fe447fd41f8a110c730c318cabe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/45e24fe447fd41f8a110c730c318cabe 2023-04-22 14:54:09,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/98d90cf740654631895c9498b8630251 2023-04-22 14:54:09,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8ba6214828d74071861f9d22c6c85f6f 2023-04-22 14:54:09,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e1c92eb4a92041c397eaedc503d5e403 2023-04-22 14:54:09,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/9ed1ddaa25c943ed887d47f3ef0cdf89 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/9ed1ddaa25c943ed887d47f3ef0cdf89 2023-04-22 14:54:09,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/5a4c7dcdf87e47bbb5dfc163b911196c 2023-04-22 14:54:09,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/50680bdeb61d4da7a3e04cfd57990762 2023-04-22 14:54:09,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/707af0914e6b4831b6cfb06447b40327 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/707af0914e6b4831b6cfb06447b40327 2023-04-22 14:54:09,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0d323a1fa51a41b3be518e39756c0391 2023-04-22 14:54:09,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/4835b2dea02a4493bed0f4829add2bec 2023-04-22 14:54:09,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/b235918ae5eb4ee394b394e9187439d7 2023-04-22 14:54:09,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/205a2acde41344caa28a1da37f54ef9c 2023-04-22 14:54:09,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/499ce98e31d042319b816dba9b52e17a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/499ce98e31d042319b816dba9b52e17a 2023-04-22 14:54:09,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/15531ed47baa459db74ae6b688c66dc0 2023-04-22 14:54:09,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/54dae5872d5e41d396743f4753c9a95b 2023-04-22 14:54:09,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/a978aa2b89504d0db6eede5313410adb 2023-04-22 14:54:09,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/401a29bac981453e88c36c9c121badbe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/401a29bac981453e88c36c9c121badbe 2023-04-22 14:54:09,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/e82fe54b78a3457584a16f2413a2caf3 2023-04-22 14:54:09,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/8c0cc45469e04cfd9b6a637ce145ff35 2023-04-22 14:54:09,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/f9bbcccde1a849ac82a18444bdb4dbbe 2023-04-22 14:54:09,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3f44224854f142d8a8cf7f995f75113e 2023-04-22 14:54:09,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c87a1ac7d65e434fad80ba3d3f60ce1a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c87a1ac7d65e434fad80ba3d3f60ce1a 2023-04-22 14:54:09,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/c29e3d5711974ce59f68e3dd14fef5e3 2023-04-22 14:54:09,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/660e78be1077424fa54c5b3e069ab4de 2023-04-22 14:54:09,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/0a837c5a7a534e968bcd1db933bc4b07 2023-04-22 14:54:09,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/56110c8a5a314243a0148efbfa7d9e6a 2023-04-22 14:54:09,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/3ace5f19360b44c891db70952f6b9fb8 2023-04-22 14:54:09,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8e3f2f292f65440a96f6be279153718b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d6ea0915a6a64ece980e61d4e158115e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0bdd4b943adf42f6acb35fb75c96903d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/440f762377f64d159eac05009fdb32ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9fde884d7dd84d5a8938512a3b0a9f9b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8b5f8ca5fae443c79744097451f1a07c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e636bf58707c4e9690ce5a8d0b87900d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb] to archive 2023-04-22 14:54:09,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:09,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/ec76bcd7165e4b66878284463df541fa 2023-04-22 14:54:09,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/086dd856175843be9125d2a8bde5d590 2023-04-22 14:54:09,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e78d168a67cd4b698dec50dda7a7d557 2023-04-22 14:54:09,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8e3f2f292f65440a96f6be279153718b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8e3f2f292f65440a96f6be279153718b 2023-04-22 14:54:09,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/7c4e0f9f1f6140daacc695f95672260f 2023-04-22 14:54:09,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/46057398faf0432eb4a2d6f124a59fff 2023-04-22 14:54:09,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d6ea0915a6a64ece980e61d4e158115e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d6ea0915a6a64ece980e61d4e158115e 2023-04-22 14:54:09,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/b5d1e330e6e54365848985e0a720aa18 2023-04-22 14:54:09,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0cb446f56b044f34b7c61edfa4306aed 2023-04-22 14:54:09,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0bdd4b943adf42f6acb35fb75c96903d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0bdd4b943adf42f6acb35fb75c96903d 2023-04-22 14:54:09,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/192624ff5c544e61b1c9bbe5edc81d20 2023-04-22 14:54:09,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/961d3be9bf944fffa19503d5c5b6b051 2023-04-22 14:54:09,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/440f762377f64d159eac05009fdb32ab to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/440f762377f64d159eac05009fdb32ab 2023-04-22 14:54:09,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8a6a6fe5e3764e75ad990b1b86727d2e 2023-04-22 14:54:09,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/dc04b5ca436644b2be42c1c0e917aa09 2023-04-22 14:54:09,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/20ab8bdc41834eac9566aaa7b63ff295 2023-04-22 14:54:09,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/1dcad337eba5481fb21552b90afe8eda 2023-04-22 14:54:09,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9fde884d7dd84d5a8938512a3b0a9f9b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9fde884d7dd84d5a8938512a3b0a9f9b 2023-04-22 14:54:09,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/33cf074b07cd4d7e84c9420bd5fe7928 2023-04-22 14:54:09,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/3f17be24a6194de3829af966c93cff32 2023-04-22 14:54:09,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/406ca7efb48d40bfbbd56ddf29fd3127 2023-04-22 14:54:09,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8b5f8ca5fae443c79744097451f1a07c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8b5f8ca5fae443c79744097451f1a07c 2023-04-22 14:54:09,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/79f82d07d09f4a6fac1f71a481ac8361 2023-04-22 14:54:09,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/0f46efbe9f9a42b6b6b3cbed67b07be7 2023-04-22 14:54:09,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/6ad935ace09443ee9a4c6291d4dd62c0 2023-04-22 14:54:09,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/f843915f511e4da1a733782c83bc1162 2023-04-22 14:54:09,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e636bf58707c4e9690ce5a8d0b87900d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/e636bf58707c4e9690ce5a8d0b87900d 2023-04-22 14:54:09,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/9d96df179519497b9c3b5ce34a99c81e 2023-04-22 14:54:09,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/4744837cbebb408b955c840baa134a92 2023-04-22 14:54:09,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/8ef09fb5a1d14d84ab8e555a8e86d8fa 2023-04-22 14:54:09,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/d50b56e6b24f49939fc69c993faddf9f 2023-04-22 14:54:09,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/00066833dace496bb970ec0ce0da984a 2023-04-22 14:54:09,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/81067bbc660640d19af88e643882fdcb 2023-04-22 14:54:09,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6963ddc56a384e23840da85e13294d80, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c483d59d8c8a4d98b48549ad1b623e1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/f6ef852b218c46ef8154a5792049f8e5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/09260903888646e39e74b24bd26ad880, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/d623f15705ca4984af29165150c14226, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/a1a9afdfbe3540bd8ce2ac8050cbba31, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/9060f90211c54d9a94d95648c252f3ef, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942] to archive 2023-04-22 14:54:09,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:09,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/72e4798ba4ef4e90a1ca84c8bd96b2b5 2023-04-22 14:54:09,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7815d728763e46fdb24921578b8d5e31 2023-04-22 14:54:09,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6abc8233737940d8a92df884361ee072 2023-04-22 14:54:09,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6963ddc56a384e23840da85e13294d80 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6963ddc56a384e23840da85e13294d80 2023-04-22 14:54:09,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/092da0137d0440e18bd331da5055826e 2023-04-22 14:54:09,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1b3527920b68433786760cc0baed2be8 2023-04-22 14:54:09,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c483d59d8c8a4d98b48549ad1b623e1c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c483d59d8c8a4d98b48549ad1b623e1c 2023-04-22 14:54:09,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/60a4eebc923c4d0db09d1bb27f324032 2023-04-22 14:54:09,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/92d36ac484ee439d96166340862e29f9 2023-04-22 14:54:09,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/f6ef852b218c46ef8154a5792049f8e5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/f6ef852b218c46ef8154a5792049f8e5 2023-04-22 14:54:09,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c540700feafd418c8876d078ad2c2de1 2023-04-22 14:54:09,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/27e967a0f86842c9a1ea7ffc0b6b7927 2023-04-22 14:54:09,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/09260903888646e39e74b24bd26ad880 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/09260903888646e39e74b24bd26ad880 2023-04-22 14:54:09,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e1e12018fc54fcda66ab39b7c77ebe3 2023-04-22 14:54:09,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/ef0f040fb4a349a5bf854e6fb68f82ed 2023-04-22 14:54:09,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/6e66a44296a349868a5b2d2bea9afcab 2023-04-22 14:54:09,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/214fe8a85e214106ac4e665f71788a34 2023-04-22 14:54:09,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/d623f15705ca4984af29165150c14226 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/d623f15705ca4984af29165150c14226 2023-04-22 14:54:09,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/79f915e950bf483cbe2dfb9f09a32a54 2023-04-22 14:54:09,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/4f54d36d0c234492af47a2560810331b 2023-04-22 14:54:09,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/306f0c6046574c00a3cc095347a7165c 2023-04-22 14:54:09,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/a1a9afdfbe3540bd8ce2ac8050cbba31 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/a1a9afdfbe3540bd8ce2ac8050cbba31 2023-04-22 14:54:09,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/1f81dc4ad5564dd5bf6393ce903201a9 2023-04-22 14:54:09,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7a9da98be3284b8d80fade441b6a8f11 2023-04-22 14:54:09,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/c3d2d220c71c4c578fc21a1cc03b1315 2023-04-22 14:54:09,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/863e92aec9334369a542739bd25e6745 2023-04-22 14:54:09,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/9060f90211c54d9a94d95648c252f3ef to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/9060f90211c54d9a94d95648c252f3ef 2023-04-22 14:54:09,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e42f4c411fa64d51be994c2119a5c00d 2023-04-22 14:54:09,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/08602eb04c1345d0815fcda008b09e57 2023-04-22 14:54:09,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/e7bfb31dec7d4afabd6d00191601ea8c 2023-04-22 14:54:09,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/0b785acbcc35406db3710418db57dcac 2023-04-22 14:54:09,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/205b0fec999742c4bd8e75e57546488c 2023-04-22 14:54:09,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/cf2105a71351410398acafcca078d942 2023-04-22 14:54:09,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/recovered.edits/760.seqid, newMaxSeqId=760, maxSeqId=1 2023-04-22 14:54:09,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52. 2023-04-22 14:54:09,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 58845d2ad95f1e691d0bdc0dfabebf52: 2023-04-22 14:54:09,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:09,619 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=58845d2ad95f1e691d0bdc0dfabebf52, regionState=CLOSED 2023-04-22 14:54:09,619 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175249619"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175249619"}]},"ts":"1682175249619"} 2023-04-22 14:54:09,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-22 14:54:09,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 58845d2ad95f1e691d0bdc0dfabebf52, server=jenkins-hbase4.apache.org,32935,1682175218140 in 1.6030 sec 2023-04-22 14:54:09,623 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-22 14:54:09,623 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=58845d2ad95f1e691d0bdc0dfabebf52, UNASSIGN in 1.6080 sec 2023-04-22 14:54:09,624 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175249624"}]},"ts":"1682175249624"} 2023-04-22 14:54:09,625 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:54:09,627 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:54:09,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6280 sec 2023-04-22 14:54:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-22 14:54:10,114 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-22 14:54:10,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:54:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,127 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,129 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 14:54:10,133 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:10,137 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/recovered.edits] 2023-04-22 14:54:10,140 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/274286b8863a4a1084e764aa0cda40dd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/274286b8863a4a1084e764aa0cda40dd 2023-04-22 14:54:10,141 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/d94917d518974534ac50d3436a7870ef to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/A/d94917d518974534ac50d3436a7870ef 2023-04-22 14:54:10,144 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/68f9a4668ae948208bbc1b7d0982b778 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/68f9a4668ae948208bbc1b7d0982b778 2023-04-22 14:54:10,145 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/c1542f26d4e145daa6924f690fbb8e85 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/B/c1542f26d4e145daa6924f690fbb8e85 2023-04-22 14:54:10,148 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/16b06ccd0ad246ec82ec48e11223b4b0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/16b06ccd0ad246ec82ec48e11223b4b0 2023-04-22 14:54:10,149 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7536a9ba1db7413ca1f94879a7d83a05 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/C/7536a9ba1db7413ca1f94879a7d83a05 2023-04-22 14:54:10,153 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/recovered.edits/760.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52/recovered.edits/760.seqid 2023-04-22 14:54:10,154 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/58845d2ad95f1e691d0bdc0dfabebf52 2023-04-22 14:54:10,154 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:54:10,156 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,177 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:54:10,222 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:54:10,223 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,223 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:54:10,224 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175250223"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:10,226 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:54:10,227 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 58845d2ad95f1e691d0bdc0dfabebf52, NAME => 'TestAcidGuarantees,,1682175221512.58845d2ad95f1e691d0bdc0dfabebf52.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:54:10,227 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:54:10,227 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175250227"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:10,229 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:54:10,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 14:54:10,232 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,233 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 112 msec 2023-04-22 14:54:10,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-22 14:54:10,433 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-22 14:54:10,451 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-578014217_15 at /127.0.0.1:37506 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:32935-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d5c3a4f-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d5c3a4f-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-578014217_15 at /127.0.0.1:37602 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d5c3a4f-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7d5c3a4f-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=546 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=450 (was 316) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=12552 (was 13333) 2023-04-22 14:54:10,463 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=546, MaxFileDescriptor=60000, SystemLoadAverage=450, ProcessCount=172, AvailableMemoryMB=12551 2023-04-22 14:54:10,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:54:10,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:54:10,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:10,469 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:54:10,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-22 14:54:10,470 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:54:10,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:10,472 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:10,473 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 empty. 2023-04-22 14:54:10,473 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:10,473 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:54:10,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:10,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:10,883 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:10,886 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8beb3d080ed1cdf928164eb126128b28, NAME => 'TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:54:11,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8beb3d080ed1cdf928164eb126128b28, disabling compactions & flushes 2023-04-22 14:54:11,295 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. after waiting 0 ms 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,295 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,295 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:11,299 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:54:11,300 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175251300"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175251300"}]},"ts":"1682175251300"} 2023-04-22 14:54:11,302 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:54:11,303 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:54:11,303 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175251303"}]},"ts":"1682175251303"} 2023-04-22 14:54:11,304 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:54:11,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, ASSIGN}] 2023-04-22 14:54:11,309 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, ASSIGN 2023-04-22 14:54:11,310 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:54:11,461 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:11,462 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175251461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175251461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175251461"}]},"ts":"1682175251461"} 2023-04-22 14:54:11,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:11,621 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8beb3d080ed1cdf928164eb126128b28, NAME => 'TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:11,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:11,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,623 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,624 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A 2023-04-22 14:54:11,624 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A 2023-04-22 14:54:11,625 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:11,625 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName A 2023-04-22 14:54:11,625 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:11,626 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,627 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B 2023-04-22 14:54:11,627 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B 2023-04-22 14:54:11,627 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:11,627 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName B 2023-04-22 14:54:11,628 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:11,628 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,629 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C 2023-04-22 14:54:11,629 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C 2023-04-22 14:54:11,629 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:11,630 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName C 2023-04-22 14:54:11,630 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:11,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:54:11,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:11,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:54:11,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8beb3d080ed1cdf928164eb126128b28; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67469398, jitterRate=0.005372375249862671}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:54:11,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:11,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., pid=18, masterSystemTime=1682175251616 2023-04-22 14:54:11,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:11,639 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:11,639 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175251639"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175251639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175251639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175251639"}]},"ts":"1682175251639"} 2023-04-22 14:54:11,642 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-22 14:54:11,643 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 in 177 msec 2023-04-22 14:54:11,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-22 14:54:11,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, ASSIGN in 336 msec 2023-04-22 14:54:11,645 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:54:11,646 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175251646"}]},"ts":"1682175251646"} 2023-04-22 14:54:11,647 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:54:11,649 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:54:11,650 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1830 sec 2023-04-22 14:54:12,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-22 14:54:12,577 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-22 14:54:12,578 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:12,584 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:12,586 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:12,590 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:12,592 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:54:12,596 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:54:12,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:54:12,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:54:12,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:12,637 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:54:12,638 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:12,638 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:54:12,643 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 14:54:12,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, REOPEN/MOVE}] 2023-04-22 14:54:12,663 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, REOPEN/MOVE 2023-04-22 14:54:12,664 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:12,664 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175252664"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175252664"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175252664"}]},"ts":"1682175252664"} 2023-04-22 14:54:12,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:12,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:12,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8beb3d080ed1cdf928164eb126128b28, disabling compactions & flushes 2023-04-22 14:54:12,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:12,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:12,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. after waiting 0 ms 2023-04-22 14:54:12,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:12,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 14:54:12,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:12,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:12,825 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8beb3d080ed1cdf928164eb126128b28 to self. 2023-04-22 14:54:12,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:12,827 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=CLOSED 2023-04-22 14:54:12,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175252827"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175252827"}]},"ts":"1682175252827"} 2023-04-22 14:54:12,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-22 14:54:12,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 in 163 msec 2023-04-22 14:54:12,831 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=true 2023-04-22 14:54:12,982 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:12,982 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175252981"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175252981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175252981"}]},"ts":"1682175252981"} 2023-04-22 14:54:12,983 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:13,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8beb3d080ed1cdf928164eb126128b28, NAME => 'TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:13,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:13,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,143 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,143 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A 2023-04-22 14:54:13,144 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A 2023-04-22 14:54:13,144 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:13,146 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName A 2023-04-22 14:54:13,148 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:13,149 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,150 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B 2023-04-22 14:54:13,150 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B 2023-04-22 14:54:13,150 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:13,150 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName B 2023-04-22 14:54:13,151 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:13,151 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,152 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C 2023-04-22 14:54:13,152 DEBUG [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C 2023-04-22 14:54:13,152 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:13,152 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8beb3d080ed1cdf928164eb126128b28 columnFamilyName C 2023-04-22 14:54:13,153 INFO [StoreOpener-8beb3d080ed1cdf928164eb126128b28-1] regionserver.HStore(310): Store=8beb3d080ed1cdf928164eb126128b28/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:13,153 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:54:13,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8beb3d080ed1cdf928164eb126128b28; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59641893, jitterRate=-0.11126653850078583}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:54:13,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:13,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., pid=23, masterSystemTime=1682175253135 2023-04-22 14:54:13,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,161 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,161 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175253161"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175253161"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175253161"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175253161"}]},"ts":"1682175253161"} 2023-04-22 14:54:13,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-22 14:54:13,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 in 179 msec 2023-04-22 14:54:13,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-22 14:54:13,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, REOPEN/MOVE in 502 msec 2023-04-22 14:54:13,168 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-22 14:54:13,168 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 525 msec 2023-04-22 14:54:13,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 556 msec 2023-04-22 14:54:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 14:54:13,178 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,183 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,184 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,188 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,189 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,192 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,193 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,199 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,199 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,203 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,203 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,207 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,207 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,211 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,211 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,216 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,216 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:13,220 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:13,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:13,232 DEBUG [hconnection-0x57477399-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,232 DEBUG [hconnection-0x547fdd2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,232 DEBUG [hconnection-0x53a9d751-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,232 DEBUG [hconnection-0x46b865b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,233 DEBUG [hconnection-0x759d2f36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,234 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,235 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,236 DEBUG [hconnection-0x2d9ad0fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,238 DEBUG [hconnection-0x12a8ccb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,243 DEBUG [hconnection-0x2a6d4b5f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,242 DEBUG [hconnection-0x37e825a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:13,241 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,241 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,245 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,244 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,262 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,262 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:13,266 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:13,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:13,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:13,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:13,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:13,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:13,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:13,276 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,276 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:13,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:13,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:13,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:13,280 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:13,280 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:13,280 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:13,281 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,281 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:13,282 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:13,316 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:13,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:13,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682175253329, max timestamp=1682175253349Num uniques 859; 2023-04-22 14:54:13,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682175253329, max timestamp=1682175253349Num uniques 859; 2023-04-22 14:54:13,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 14:54:13,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 14:54:13,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175253343, max timestamp=1682175253349Num uniques 859; 2023-04-22 14:54:13,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682175253366, max timestamp=1682175253374Num uniques 1218; 2023-04-22 14:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682175253366, max timestamp=1682175253374Num uniques 1218; 2023-04-22 14:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=16848, suffix heap size=154640,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=368 2023-04-22 14:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=16848, suffix heap size=154640,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=368 2023-04-22 14:54:13,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7684, totalHeapSize=17712, min timestamp=1682175253371, max timestamp=1682175253379Num uniques 1018; 2023-04-22 14:54:13,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46614, new segment data size=7684, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:54:13,409 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ea68a4d5bbab4c0988a91247b29bfcb8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ea68a4d5bbab4c0988a91247b29bfcb8 2023-04-22 14:54:13,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175313404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175313405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175253390, max timestamp=1682175253413Num uniques 1227; 2023-04-22 14:54:13,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175253390, max timestamp=1682175253413Num uniques 1027; 2023-04-22 14:54:13,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-22 14:54:13,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175253390, max timestamp=1682175253413Num uniques 1227; 2023-04-22 14:54:13,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-22 14:54:13,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-22 14:54:13,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682175253423, max timestamp=1682175253435Num uniques 1086; 2023-04-22 14:54:13,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682175253423, max timestamp=1682175253435Num uniques 1086; 2023-04-22 14:54:13,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-22 14:54:13,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-22 14:54:13,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682175253423, max timestamp=1682175253435Num uniques 1086; 2023-04-22 14:54:13,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-22 14:54:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175313461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175313461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=495, cellSize=22670, totalHeapSize=51720, min timestamp=1682175253446, max timestamp=1682175253461Num uniques 1195; 2023-04-22 14:54:13,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=495, cellSize=22670, totalHeapSize=51720, min timestamp=1682175253446, max timestamp=1682175253461Num uniques 1195; 2023-04-22 14:54:13,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=22670, suffix heap size=152248,new segment heap size=51720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=495 2023-04-22 14:54:13,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=22670, suffix heap size=152248,new segment heap size=51720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=495 2023-04-22 14:54:13,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682175253451, max timestamp=1682175253461Num uniques 1195; 2023-04-22 14:54:13,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=6870, suffix heap size=152248,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=150 2023-04-22 14:54:13,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175313518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175313520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175313522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175313567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175313571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175313626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175313626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175313723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175313771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175313774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175313829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175313836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,852 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=11, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/ee574aa943e14586a15b83cf180875f3 2023-04-22 14:54:13,892 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/72de8988de754cf6a13657fe4c0a00be 2023-04-22 14:54:13,923 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/2364cfe6ac484910b77a75871e4a3824 2023-04-22 14:54:13,928 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/ee574aa943e14586a15b83cf180875f3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3 2023-04-22 14:54:13,934 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3, entries=150, sequenceid=11, filesize=25.1 K 2023-04-22 14:54:13,936 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/72de8988de754cf6a13657fe4c0a00be as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be 2023-04-22 14:54:13,944 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be, entries=150, sequenceid=11, filesize=11.6 K 2023-04-22 14:54:13,945 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/2364cfe6ac484910b77a75871e4a3824 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824 2023-04-22 14:54:13,957 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824, entries=150, sequenceid=11, filesize=11.6 K 2023-04-22 14:54:13,958 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=159.00 KB/162820 for 8beb3d080ed1cdf928164eb126128b28 in 676ms, sequenceid=11, compaction requested=false 2023-04-22 14:54:13,958 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 14:54:13,959 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:13,959 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:13,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:13,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:13,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:13,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:13,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@225cbe4d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:13,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:13,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,966 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:13,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:13,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:13,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:13,968 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:13,968 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:13,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,968 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:13,972 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:13,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:13,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:13,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:13,972 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:13,975 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,975 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,975 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:13,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:13,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,976 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:13,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:13,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:13,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:13,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:13,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:13,981 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:13,981 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:13,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:13,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:13,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:14,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:14,034 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=165.71 KB heapSize=457.90 KB 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:14,035 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:14,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175314049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175314077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,081 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175314079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,086 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ab947ac2ce49442d8831c9c2d90b4749 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ab947ac2ce49442d8831c9c2d90b4749 2023-04-22 14:54:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,104 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,126 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abd18de[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,131 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,131 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175314134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,139 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175314141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,145 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,146 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175314151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,162 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,163 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,163 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,263 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,270 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,286 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,287 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,287 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bee2834[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,288 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,288 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,290 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,290 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,290 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,292 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,292 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,292 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,293 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,297 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,297 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,299 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,299 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,299 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,305 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,305 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175314353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,405 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,410 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,412 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,412 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,422 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,422 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,422 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,422 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,426 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6cd172[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,428 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,428 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,429 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,431 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,434 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,435 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,435 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,435 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,435 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,437 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,437 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,437 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,438 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,439 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,444 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,445 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,445 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,445 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,445 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,445 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,520 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/1c147581d2544d4790bb90c54f63d30f 2023-04-22 14:54:14,545 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,550 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,550 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,556 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7985ae24[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,558 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,558 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,559 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,562 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,565 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,565 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,567 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,567 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,567 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,571 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,572 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,572 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,572 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175314582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175314588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175314638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175314646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:14,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175314658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,673 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,680 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,681 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,681 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,690 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,690 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,690 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,691 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,700 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,700 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,700 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,700 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,702 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5c3222[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,707 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,707 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,707 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,708 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,709 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,712 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,712 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,719 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,724 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,724 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,724 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,724 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:14,824 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,828 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,828 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,828 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,831 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,833 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,833 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,833 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,833 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d90ef5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,841 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,841 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,841 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,848 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,848 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,850 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,850 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,850 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:14,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:14,857 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:14,857 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:14,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,858 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,858 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,858 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,858 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/e480580b39f648acaf110918f745f988 2023-04-22 14:54:14,957 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:14,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:14,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:14,965 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:14,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:14,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:14,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:14,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:14,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:14,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:14,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:14,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:14,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:14,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,981 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,981 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2966b6fd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:14,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:14,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,984 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:14,984 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,984 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:14,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:14,992 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:14,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:14,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:14,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:14,992 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:14,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:14,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:14,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,005 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,005 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:15,105 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,111 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,134 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,134 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,134 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc6d97b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,137 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,139 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,140 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,140 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,147 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,152 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,152 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:15,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175315160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,162 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:54:15,253 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,259 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,267 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473129c3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,280 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,280 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,283 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,284 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,287 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:15,287 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,288 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,287 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,288 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,288 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,288 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.95 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/e3908b47225e4d8c848de96d7c55a8fb 2023-04-22 14:54:15,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/1c147581d2544d4790bb90c54f63d30f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f 2023-04-22 14:54:15,388 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f, entries=200, sequenceid=117, filesize=31.9 K 2023-04-22 14:54:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,392 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/e480580b39f648acaf110918f745f988 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988 2023-04-22 14:54:15,394 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 14:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/e3908b47225e4d8c848de96d7c55a8fb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb 2023-04-22 14:54:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,401 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,401 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,401 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,401 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 14:54:15,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~165.71 KB/169690, heapSize ~457.85 KB/468840, currentSize=20.13 KB/20610 for 8beb3d080ed1cdf928164eb126128b28 in 1373ms, sequenceid=117, compaction requested=false 2023-04-22 14:54:15,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:15,410 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea63976[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,412 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,412 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,414 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,414 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,414 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,415 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,418 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,419 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,419 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,419 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,419 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,420 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,420 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,420 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,421 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,423 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,428 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,427 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,428 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,528 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,533 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,535 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,536 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,536 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,548 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,549 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:15,549 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:15,591 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223323b799cbcf45489eaa305b133f8e82 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223323b799cbcf45489eaa305b133f8e82 2023-04-22 14:54:15,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:15,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,608 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=123, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/08f9463c49c54ec48cf38d8ca04a9e65 2023-04-22 14:54:15,634 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/12e5873a95854835852cced33ed47d2a 2023-04-22 14:54:15,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175255628, max timestamp=1682175255639Num uniques 859; 2023-04-22 14:54:15,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:15,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175255628, max timestamp=1682175255639Num uniques 859; 2023-04-22 14:54:15,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:15,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175255628, max timestamp=1682175255639Num uniques 859; 2023-04-22 14:54:15,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:15,670 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/82cf5011c266447995e19c286dfa6caa 2023-04-22 14:54:15,677 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/08f9463c49c54ec48cf38d8ca04a9e65 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175255664, max timestamp=1682175255677Num uniques 1068; 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175255664, max timestamp=1682175255677Num uniques 1068; 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175255664, max timestamp=1682175255677Num uniques 1068; 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-22 14:54:15,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-22 14:54:15,686 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65, entries=100, sequenceid=123, filesize=18.4 K 2023-04-22 14:54:15,687 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/12e5873a95854835852cced33ed47d2a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a 2023-04-22 14:54:15,694 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a, entries=100, sequenceid=123, filesize=9.3 K 2023-04-22 14:54:15,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/82cf5011c266447995e19c286dfa6caa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa 2023-04-22 14:54:15,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa, entries=100, sequenceid=123, filesize=9.3 K 2023-04-22 14:54:15,713 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=60.38 KB/61830 for 8beb3d080ed1cdf928164eb126128b28 in 164ms, sequenceid=123, compaction requested=true 2023-04-22 14:54:15,713 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:15,713 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:15,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=80.51 KB heapSize=220.08 KB 2023-04-22 14:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:15,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:15,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:15,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:15,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:15,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:15,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62eba901[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,720 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,720 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,725 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:15,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,729 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,729 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,729 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,729 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,729 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,734 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304228f0bc9e86fbb4e85993d0b9f8f49a94e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f0bc9e86fbb4e85993d0b9f8f49a94e 2023-04-22 14:54:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,739 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,743 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,743 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175315739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175315740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175315741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175315744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,750 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/f7fe989d1a984447beb2cef53577899e 2023-04-22 14:54:15,843 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175315846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,848 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175315846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175315846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,849 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,849 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,851 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,853 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,853 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,853 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,853 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:15,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175315851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,856 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292da926[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,858 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,858 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,864 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,864 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,867 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,872 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,872 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,872 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:15,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:15,873 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,873 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,973 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:15,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:15,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:15,979 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:15,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:15,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:15,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:15,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:15,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:15,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:15,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:15,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:15,985 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:15,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671cf2f1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:15,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,988 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:15,988 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,988 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:15,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:15,994 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:15,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:15,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:15,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:15,994 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:15,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:15,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,009 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:16,009 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175316050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175316050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175316051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175316056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,109 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,114 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:16,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28be0aed[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,123 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:16,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:16,128 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:16,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:16,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:16,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:16,128 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:16,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,134 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,140 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,140 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175316164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/a7d5123958b8408eab28990704e079a7 2023-04-22 14:54:16,240 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,244 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,250 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,250 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,250 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:16,250 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,253 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,253 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,253 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17220d5a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,258 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:16,261 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:16,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:16,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:16,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:16,261 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:16,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,264 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,269 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,270 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:16,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175316352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175316353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175316354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175316359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,370 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,373 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,375 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:16,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:16,377 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5382f201[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,385 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:16,387 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:16,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:16,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:16,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:16,387 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:16,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,396 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,396 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,396 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,396 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,497 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,500 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:16,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:16,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530919fa[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,513 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:16,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:16,517 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:16,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:16,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:16,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:16,517 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:16,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,520 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,521 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:16,526 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,527 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/82107e05b2fb4aec81ee2bc26828d71a 2023-04-22 14:54:16,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/f7fe989d1a984447beb2cef53577899e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e 2023-04-22 14:54:16,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e, entries=300, sequenceid=169, filesize=45.8 K 2023-04-22 14:54:16,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/a7d5123958b8408eab28990704e079a7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7 2023-04-22 14:54:16,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7, entries=150, sequenceid=169, filesize=11.8 K 2023-04-22 14:54:16,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/82107e05b2fb4aec81ee2bc26828d71a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a 2023-04-22 14:54:16,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a, entries=150, sequenceid=169, filesize=11.8 K 2023-04-22 14:54:16,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~239.95 KB/245712, currentSize=93.93 KB/96180 for 8beb3d080ed1cdf928164eb126128b28 in 895ms, sequenceid=169, compaction requested=true 2023-04-22 14:54:16,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:16,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:16,609 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:16,609 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:16,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:16,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:16,610 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:16,610 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:16,610 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/A is initiating minor compaction (all files) 2023-04-22 14:54:16,610 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/B is initiating minor compaction (all files) 2023-04-22 14:54:16,610 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/A in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,610 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/B in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,611 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=44.3 K 2023-04-22 14:54:16,611 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=121.2 K 2023-04-22 14:54:16,611 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 72de8988de754cf6a13657fe4c0a00be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682175253247 2023-04-22 14:54:16,611 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ee574aa943e14586a15b83cf180875f3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682175253247 2023-04-22 14:54:16,611 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e480580b39f648acaf110918f745f988, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175253500 2023-04-22 14:54:16,611 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1c147581d2544d4790bb90c54f63d30f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175253500 2023-04-22 14:54:16,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 12e5873a95854835852cced33ed47d2a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175254040 2023-04-22 14:54:16,612 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 08f9463c49c54ec48cf38d8ca04a9e65, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175254040 2023-04-22 14:54:16,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a7d5123958b8408eab28990704e079a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:16,612 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f7fe989d1a984447beb2cef53577899e, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:16,623 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#B#compaction#118 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:16,629 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,635 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#A#compaction#117 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:16,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,639 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,641 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:16,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,655 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,655 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,655 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,656 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:16,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,671 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/b7cf5bf5598b43c9b41abef6bb6ed66f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b7cf5bf5598b43c9b41abef6bb6ed66f 2023-04-22 14:54:16,678 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/B of 8beb3d080ed1cdf928164eb126128b28 into b7cf5bf5598b43c9b41abef6bb6ed66f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:16,678 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:16,678 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/B, priority=12, startTime=1682175256609; duration=0sec 2023-04-22 14:54:16,678 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:16,678 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:16,680 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:16,680 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/C is initiating minor compaction (all files) 2023-04-22 14:54:16,680 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/C in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,680 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=44.3 K 2023-04-22 14:54:16,681 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2364cfe6ac484910b77a75871e4a3824, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682175253247 2023-04-22 14:54:16,682 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e3908b47225e4d8c848de96d7c55a8fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175253500 2023-04-22 14:54:16,682 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 82cf5011c266447995e19c286dfa6caa, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175254040 2023-04-22 14:54:16,683 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 82107e05b2fb4aec81ee2bc26828d71a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:16,689 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b5dadb8929b8445e8d1322411b262b1d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b5dadb8929b8445e8d1322411b262b1d 2023-04-22 14:54:16,719 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/5db98feed5504386a1571c962bd62d3c 2023-04-22 14:54:16,720 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#C#compaction#120 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:16,746 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/7ced2f25e6e74c3894930795b758a343 2023-04-22 14:54:16,752 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/60de038d47d34900b4433791d1a1c2bd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/60de038d47d34900b4433791d1a1c2bd 2023-04-22 14:54:16,761 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/C of 8beb3d080ed1cdf928164eb126128b28 into 60de038d47d34900b4433791d1a1c2bd(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:16,761 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:16,761 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/C, priority=12, startTime=1682175256609; duration=0sec 2023-04-22 14:54:16,761 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:16,785 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/5c0c39b2574f4abd80040362d1516178 2023-04-22 14:54:16,801 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/5db98feed5504386a1571c962bd62d3c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c 2023-04-22 14:54:16,807 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c, entries=150, sequenceid=186, filesize=25.3 K 2023-04-22 14:54:16,808 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/7ced2f25e6e74c3894930795b758a343 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343 2023-04-22 14:54:16,821 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343, entries=150, sequenceid=186, filesize=11.8 K 2023-04-22 14:54:16,828 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/5c0c39b2574f4abd80040362d1516178 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178 2023-04-22 14:54:16,841 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178, entries=150, sequenceid=186, filesize=11.8 K 2023-04-22 14:54:16,842 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 8beb3d080ed1cdf928164eb126128b28 in 186ms, sequenceid=186, compaction requested=false 2023-04-22 14:54:16,842 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:16,842 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,843 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,848 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,848 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b388814[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,852 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,856 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:16,859 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:16,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:16,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:16,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:16,859 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:16,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:16,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:16,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:16,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:16,869 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:16,870 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:16,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:16,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 14:54:16,904 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:16,904 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,904 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:16,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:16,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:16,970 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:16,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:16,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:16,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682175256956, max timestamp=1682175256973Num uniques 859; 2023-04-22 14:54:16,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 14:54:16,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682175256956, max timestamp=1682175256973Num uniques 859; 2023-04-22 14:54:16,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 14:54:16,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682175256956, max timestamp=1682175256973Num uniques 859; 2023-04-22 14:54:16,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-22 14:54:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:16,980 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:16,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:16,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:16,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,982 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:16,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:16,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:16,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:16,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:16,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:16,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:16,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:16,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:16,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:16,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:16,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:16,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,992 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,992 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:16,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:16,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c771cf4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:16,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:16,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:16,997 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,997 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:16,998 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,998 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:16,998 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:16,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:16,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175316996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:16,998 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,001 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,002 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,002 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,002 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175316998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175316998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175316998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,012 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,013 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,013 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:17,013 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175317100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175317105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175317105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175317106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,114 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,117 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,121 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/738a10c84b8a429699a527daa408ca58 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/738a10c84b8a429699a527daa408ca58 2023-04-22 14:54:17,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,122 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,122 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,122 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,122 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a1f581[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,126 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,128 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,128 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,128 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,128 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,129 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,129 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/A of 8beb3d080ed1cdf928164eb126128b28 into 738a10c84b8a429699a527daa408ca58(size=25.4 K), total size for store is 50.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:17,130 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:17,130 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/A, priority=12, startTime=1682175256609; duration=0sec 2023-04-22 14:54:17,130 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:17,130 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,130 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,130 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,130 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,133 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,133 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,136 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,139 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,143 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,143 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,143 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,144 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,144 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,244 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,248 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,255 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,255 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,255 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,255 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,255 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,257 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,257 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,257 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fce3a6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,262 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,262 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,262 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,265 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,265 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,273 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,273 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175317306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175317309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175317311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175317311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,339 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223850cb81a9f64ddfb45e0fce7dfae140 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223850cb81a9f64ddfb45e0fce7dfae140 2023-04-22 14:54:17,373 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,377 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,377 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,377 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,377 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4bb9e5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,391 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,392 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,394 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,398 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,398 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,398 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:17,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,499 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,503 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,507 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,507 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,507 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,507 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286f8624[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,513 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,513 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,517 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,517 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,520 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,521 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:17,524 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,523 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,524 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,524 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175317610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175317613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175317614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175317615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,624 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,627 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,629 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa57355[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,644 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,644 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,646 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,648 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,651 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,651 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,652 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,746 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/06acd2d4b02849268a4ec8316df97d07 2023-04-22 14:54:17,751 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,760 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,762 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/ba7764ac30eb4de48733f119d36c891c 2023-04-22 14:54:17,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf05cd4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,774 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,774 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,776 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,778 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,779 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,783 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,787 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,787 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,887 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:17,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:17,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:17,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:17,891 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:17,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:17,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:17,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:17,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:17,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:17,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:17,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:17,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:17,903 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab542de[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:17,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:17,905 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,906 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,906 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,906 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:17,906 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:17,906 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:17,906 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:17,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:17,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:17,911 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:17,911 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:17,911 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:17,911 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:17,911 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:17,912 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,912 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:17,912 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:17,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:17,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:17,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:17,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:17,918 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:17,918 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:17,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:17,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:17,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:17,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:18,019 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:18,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:18,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:18,023 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:18,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:18,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:18,024 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,024 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,025 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:18,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:18,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,026 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:18,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:18,027 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,027 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:18,027 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:18,027 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:18,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,029 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ebfa90a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:18,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:18,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:18,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:18,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:18,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:18,043 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:18,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:18,046 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:18,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:18,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:18,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:18,046 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:18,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:18,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,052 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:18,052 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:18,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:18,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:18,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:18,052 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,053 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:18,053 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:18,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:18,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175318113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175318115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:18,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175318121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175318123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,153 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:18,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:18,157 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:18,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:18,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:18,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:18,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:18,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:18,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:18,162 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,162 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:18,162 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:18,163 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:18,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a445453[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:18,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:18,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,168 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:18,168 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,168 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:18,170 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:18,173 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:18,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:18,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:18,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:18,173 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:18,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:18,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:18,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,179 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:18,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:18,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:18,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:18,182 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,182 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:18,182 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:18,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:18,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:18,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175318187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,191 DEBUG [Thread-823] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:18,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0 2023-04-22 14:54:18,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/06acd2d4b02849268a4ec8316df97d07 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07 2023-04-22 14:54:18,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07, entries=350, sequenceid=199, filesize=52.6 K 2023-04-22 14:54:18,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/ba7764ac30eb4de48733f119d36c891c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c 2023-04-22 14:54:18,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c, entries=150, sequenceid=199, filesize=11.8 K 2023-04-22 14:54:18,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0 2023-04-22 14:54:18,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0, entries=150, sequenceid=199, filesize=11.8 K 2023-04-22 14:54:18,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 8beb3d080ed1cdf928164eb126128b28 in 1317ms, sequenceid=199, compaction requested=true 2023-04-22 14:54:18,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:18,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:18,222 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:18,222 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:18,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:18,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:18,223 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:18,223 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105790 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:18,224 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/B is initiating minor compaction (all files) 2023-04-22 14:54:18,224 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/A is initiating minor compaction (all files) 2023-04-22 14:54:18,224 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/B in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,224 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/A in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,224 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b7cf5bf5598b43c9b41abef6bb6ed66f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=35.4 K 2023-04-22 14:54:18,224 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/738a10c84b8a429699a527daa408ca58, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=103.3 K 2023-04-22 14:54:18,224 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b7cf5bf5598b43c9b41abef6bb6ed66f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:18,225 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 738a10c84b8a429699a527daa408ca58, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:18,225 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7ced2f25e6e74c3894930795b758a343, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682175255731 2023-04-22 14:54:18,225 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5db98feed5504386a1571c962bd62d3c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682175255731 2023-04-22 14:54:18,225 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ba7764ac30eb4de48733f119d36c891c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256901 2023-04-22 14:54:18,226 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 06acd2d4b02849268a4ec8316df97d07, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256885 2023-04-22 14:54:18,244 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#B#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:18,254 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#A#compaction#127 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:18,282 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:18,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:18,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:18,288 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:18,288 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,288 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:18,288 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:18,289 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,289 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,291 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,291 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:18,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:18,293 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:18,293 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:18,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:18,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:18,294 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-22 14:54:18,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:18,298 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/ce39317b9382475a9affec9dffab9953 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ce39317b9382475a9affec9dffab9953 2023-04-22 14:54:18,305 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/A of 8beb3d080ed1cdf928164eb126128b28 into ce39317b9382475a9affec9dffab9953(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:18,305 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:18,305 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/A, priority=13, startTime=1682175258221; duration=0sec 2023-04-22 14:54:18,306 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:18,306 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:18,307 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:18,307 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/C is initiating minor compaction (all files) 2023-04-22 14:54:18,307 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/C in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:18,307 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/60de038d47d34900b4433791d1a1c2bd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=35.4 K 2023-04-22 14:54:18,308 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 60de038d47d34900b4433791d1a1c2bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682175255682 2023-04-22 14:54:18,308 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5c0c39b2574f4abd80040362d1516178, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682175255731 2023-04-22 14:54:18,309 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3b1ddfa3b14e40eba7cb9b7be8daa1e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256901 2023-04-22 14:54:18,319 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#C#compaction#129 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:18,669 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/84c83974b60f439bbc185e22c3cb13df as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/84c83974b60f439bbc185e22c3cb13df 2023-04-22 14:54:18,674 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/B of 8beb3d080ed1cdf928164eb126128b28 into 84c83974b60f439bbc185e22c3cb13df(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:18,674 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:18,674 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/B, priority=13, startTime=1682175258222; duration=0sec 2023-04-22 14:54:18,674 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:18,745 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d444679f1bc8419a8c15446ff85f5c63 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d444679f1bc8419a8c15446ff85f5c63 2023-04-22 14:54:18,760 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/6984e151c5c747d880a3e0fe330ccbba as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/6984e151c5c747d880a3e0fe330ccbba 2023-04-22 14:54:18,765 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/C of 8beb3d080ed1cdf928164eb126128b28 into 6984e151c5c747d880a3e0fe330ccbba(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:18,766 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:18,766 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/C, priority=13, startTime=1682175258222; duration=0sec 2023-04-22 14:54:18,766 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:19,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:19,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:19,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175319135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175319136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175319136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175319136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,159 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=234, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/763047f0e05f459595232ba4a12b71bc 2023-04-22 14:54:19,172 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/cad98cf02667458396f76736ddb3bdc8 2023-04-22 14:54:19,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175319237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175319241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175319241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175319241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175319441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175319446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175319446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175319446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,588 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/4aa96dd89a7c4bd192fb308ed7070942 2023-04-22 14:54:19,593 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/763047f0e05f459595232ba4a12b71bc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc 2023-04-22 14:54:19,598 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc, entries=150, sequenceid=234, filesize=25.3 K 2023-04-22 14:54:19,599 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/cad98cf02667458396f76736ddb3bdc8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8 2023-04-22 14:54:19,604 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8, entries=150, sequenceid=234, filesize=11.8 K 2023-04-22 14:54:19,605 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/4aa96dd89a7c4bd192fb308ed7070942 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942 2023-04-22 14:54:19,609 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942, entries=150, sequenceid=234, filesize=11.8 K 2023-04-22 14:54:19,610 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=67.09 KB/68700 for 8beb3d080ed1cdf928164eb126128b28 in 1316ms, sequenceid=234, compaction requested=false 2023-04-22 14:54:19,610 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:19,610 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:19,610 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:19,610 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:19,610 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:19,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:19,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@538bbf33[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:19,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:19,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,615 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:19,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:19,617 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:19,621 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:19,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:19,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:19,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:19,621 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:19,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:19,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:19,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:19,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:19,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,634 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:19,634 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:19,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,734 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:19,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:19,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:19,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:19,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:19,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:19,748 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:19,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:19,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:19,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:19,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:19,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:19,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:19,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175319765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175319765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175319772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175319772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:19,784 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:19,784 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:19,784 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:19,784 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:19,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:19,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6318b42e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:19,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:19,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,791 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,792 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:19,793 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:19,793 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,793 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:19,794 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:19,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:19,798 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:19,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:19,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:19,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:19,798 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:19,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,804 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,809 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,812 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:19,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:19,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,813 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:19,813 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:19,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:19,815 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c53f8101a98c48dbb880a77afbc0ce32 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c53f8101a98c48dbb880a77afbc0ce32 2023-04-22 14:54:19,830 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/42093676e58243f0a52559020f56d709 2023-04-22 14:54:19,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175319870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175319871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175319873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:19,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175319875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,913 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:19,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:19,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:19,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:19,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:19,917 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:19,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:19,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:19,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:19,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:19,919 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:19,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:19,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:19,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:19,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:19,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:19,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:19,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:19,924 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:19,926 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:19,926 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dca6be[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:19,928 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:19,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:19,929 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,929 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:19,930 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:19,933 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:19,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:19,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:19,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:19,934 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:19,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:19,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:19,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:19,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:19,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:19,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:19,942 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:19,942 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:19,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,042 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:20,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:20,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:20,046 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,047 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:20,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:20,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:20,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:20,050 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,050 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:20,050 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:20,050 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:20,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:20,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7812963a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:20,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,055 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:20,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:20,057 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:20,060 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:20,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:20,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:20,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:20,060 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:20,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:20,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:20,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:20,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:20,066 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,066 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:20,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175320074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175320075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175320076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175320077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,167 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:20,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:20,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:20,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:20,170 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:20,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:20,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:20,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:20,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:20,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:20,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:20,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:20,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:20,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:20,176 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:20,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76eedc2c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:20,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,180 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:20,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,180 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:20,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:20,184 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:20,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:20,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:20,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:20,184 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:20,186 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,186 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,186 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:20,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,186 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:20,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:20,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:20,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:20,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:20,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:20,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,193 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:20,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:20,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/1bdf7dab4449488cbc0e6f7bb7ae45ec 2023-04-22 14:54:20,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/a0407cb72fdb43d794345ba2751f4a5f 2023-04-22 14:54:20,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/42093676e58243f0a52559020f56d709 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709 2023-04-22 14:54:20,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709, entries=200, sequenceid=251, filesize=32.1 K 2023-04-22 14:54:20,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/1bdf7dab4449488cbc0e6f7bb7ae45ec as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec 2023-04-22 14:54:20,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec, entries=150, sequenceid=251, filesize=11.8 K 2023-04-22 14:54:20,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/a0407cb72fdb43d794345ba2751f4a5f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f 2023-04-22 14:54:20,294 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:20,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:20,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:20,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f, entries=150, sequenceid=251, filesize=11.8 K 2023-04-22 14:54:20,298 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8beb3d080ed1cdf928164eb126128b28 in 553ms, sequenceid=251, compaction requested=true 2023-04-22 14:54:20,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:20,298 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:20,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:20,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:20,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:20,298 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:20,299 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:20,299 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/A is initiating minor compaction (all files) 2023-04-22 14:54:20,300 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/A in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,300 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ce39317b9382475a9affec9dffab9953, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=82.9 K 2023-04-22 14:54:20,300 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:20,300 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/B is initiating minor compaction (all files) 2023-04-22 14:54:20,300 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/B in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,300 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/84c83974b60f439bbc185e22c3cb13df, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=35.5 K 2023-04-22 14:54:20,301 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ce39317b9382475a9affec9dffab9953, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256901 2023-04-22 14:54:20,301 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 84c83974b60f439bbc185e22c3cb13df, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256901 2023-04-22 14:54:20,301 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 763047f0e05f459595232ba4a12b71bc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682175256994 2023-04-22 14:54:20,301 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting cad98cf02667458396f76736ddb3bdc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682175256994 2023-04-22 14:54:20,301 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 42093676e58243f0a52559020f56d709, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259133 2023-04-22 14:54:20,302 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1bdf7dab4449488cbc0e6f7bb7ae45ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259135 2023-04-22 14:54:20,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:20,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:20,316 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:20,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:20,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:20,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,318 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,318 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,318 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,319 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:20,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:20,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:20,322 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#B#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:20,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:20,327 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:20,328 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:20,337 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#A#compaction#136 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:20,354 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/c76be6e2e9b84ee7bef8e5f38d1bb88e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/c76be6e2e9b84ee7bef8e5f38d1bb88e 2023-04-22 14:54:20,360 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/B of 8beb3d080ed1cdf928164eb126128b28 into c76be6e2e9b84ee7bef8e5f38d1bb88e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:20,360 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:20,361 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/B, priority=13, startTime=1682175260298; duration=0sec 2023-04-22 14:54:20,361 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:20,361 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:20,362 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:20,363 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/C is initiating minor compaction (all files) 2023-04-22 14:54:20,363 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/C in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:20,363 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/6984e151c5c747d880a3e0fe330ccbba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=35.5 K 2023-04-22 14:54:20,363 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6984e151c5c747d880a3e0fe330ccbba, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682175256901 2023-04-22 14:54:20,364 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4aa96dd89a7c4bd192fb308ed7070942, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682175256994 2023-04-22 14:54:20,364 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a0407cb72fdb43d794345ba2751f4a5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259135 2023-04-22 14:54:20,369 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304221697abf239054c7aab70f7fba592b4c2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221697abf239054c7aab70f7fba592b4c2 2023-04-22 14:54:20,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:20,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:20,390 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#C#compaction#138 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:20,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175320393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175320394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175320399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175320399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,419 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/4878f7e08c8e4bdba3c6481b9ccbb699 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4878f7e08c8e4bdba3c6481b9ccbb699 2023-04-22 14:54:20,425 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/C of 8beb3d080ed1cdf928164eb126128b28 into 4878f7e08c8e4bdba3c6481b9ccbb699(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:20,425 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:20,425 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/C, priority=13, startTime=1682175260298; duration=0sec 2023-04-22 14:54:20,425 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:20,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175320500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175320501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175320504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175320504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175320702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175320703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175320706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:20,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175320708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:20,804 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/ded955c0200a446280e575ec9b7717a7 2023-04-22 14:54:20,824 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/2608c22f0346468caa8c11e5d5046f7e 2023-04-22 14:54:21,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175321006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175321007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175321009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175321011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,170 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/3c6bd4f882b94932b610cc9a4da56ef2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/3c6bd4f882b94932b610cc9a4da56ef2 2023-04-22 14:54:21,175 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/A of 8beb3d080ed1cdf928164eb126128b28 into 3c6bd4f882b94932b610cc9a4da56ef2(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:21,175 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:21,175 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/A, priority=13, startTime=1682175260298; duration=0sec 2023-04-22 14:54:21,175 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:21,236 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/8d2bd2f9c2a948138d203d4ef322ac58 2023-04-22 14:54:21,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/ded955c0200a446280e575ec9b7717a7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7 2023-04-22 14:54:21,246 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7, entries=150, sequenceid=269, filesize=25.4 K 2023-04-22 14:54:21,247 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/2608c22f0346468caa8c11e5d5046f7e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e 2023-04-22 14:54:21,252 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e, entries=150, sequenceid=269, filesize=11.9 K 2023-04-22 14:54:21,253 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/8d2bd2f9c2a948138d203d4ef322ac58 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58 2023-04-22 14:54:21,258 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58, entries=150, sequenceid=269, filesize=11.9 K 2023-04-22 14:54:21,259 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 8beb3d080ed1cdf928164eb126128b28 in 931ms, sequenceid=269, compaction requested=false 2023-04-22 14:54:21,259 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:21,259 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:21,261 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5071af1b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:21,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:21,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,265 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:21,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:21,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,267 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:21,267 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:21,267 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:21,270 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:21,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:21,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:21,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:21,270 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:21,271 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,271 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:21,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:21,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:21,272 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:21,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:21,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:21,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:21,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:21,276 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:21,276 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:21,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,276 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:21,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:21,376 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:21,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:21,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:21,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:21,381 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:21,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:21,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:21,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:21,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:21,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:21,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:21,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:21,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:21,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:21,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:21,388 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:21,388 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:21,388 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:21,389 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:21,402 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042298120d386ee44331895115978061e24a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042298120d386ee44331895115978061e24a 2023-04-22 14:54:21,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:21,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:21,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175321522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175321522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175321523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175321524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175321625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175321625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175321626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175321628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,808 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/c61cea526f2e44169d783ad7dcce859d 2023-04-22 14:54:21,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175321828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175321829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175321829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:21,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175321831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175322130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56366 deadline: 1682175322132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175322133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175322135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175322202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,204 DEBUG [Thread-823] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:22,221 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/5494a89152c74019b9f49dd54f163658 2023-04-22 14:54:22,633 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/c4a63ce332fd49fab3ef5884abcd24f4 2023-04-22 14:54:22,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56324 deadline: 1682175322636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,641 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/c61cea526f2e44169d783ad7dcce859d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d 2023-04-22 14:54:22,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175322641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56338 deadline: 1682175322641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:22,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175322641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,646 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d, entries=150, sequenceid=287, filesize=25.4 K 2023-04-22 14:54:22,647 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/5494a89152c74019b9f49dd54f163658 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658 2023-04-22 14:54:22,650 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658, entries=150, sequenceid=287, filesize=11.9 K 2023-04-22 14:54:22,652 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/c4a63ce332fd49fab3ef5884abcd24f4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4 2023-04-22 14:54:22,656 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4, entries=150, sequenceid=287, filesize=11.9 K 2023-04-22 14:54:22,656 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8beb3d080ed1cdf928164eb126128b28 in 1268ms, sequenceid=287, compaction requested=true 2023-04-22 14:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:22,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ad1afd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:22,661 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:22,661 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:22,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:22,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:22,666 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:22,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:22,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:22,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:22,667 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:22,669 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,669 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:22,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,669 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:22,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:22,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:22,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:22,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:22,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:22,674 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:22,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:22,673 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:22,674 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:22,774 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:22,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:22,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:22,784 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:22,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:22,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:22,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:22,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:22,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:22,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:22,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:22,786 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:22,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:22,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:22,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:22,789 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:22,789 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:22,789 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:22,829 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b06715bedccc419fb9cc6fdf7bdb2776 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b06715bedccc419fb9cc6fdf7bdb2776 2023-04-22 14:54:22,842 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/d2518eeb3d6c443486dca8d71492b8d1 2023-04-22 14:54:23,263 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/81e2d382051044468d02a27df9298fa1 2023-04-22 14:54:23,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:23,640 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:23,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175323653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175323654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175323655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175323655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,678 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/34a9a5c3b37c46d593a799e16feabb1c 2023-04-22 14:54:23,684 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/d2518eeb3d6c443486dca8d71492b8d1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1 2023-04-22 14:54:23,689 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1, entries=150, sequenceid=304, filesize=25.4 K 2023-04-22 14:54:23,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/81e2d382051044468d02a27df9298fa1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1 2023-04-22 14:54:23,694 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 14:54:23,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/34a9a5c3b37c46d593a799e16feabb1c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c 2023-04-22 14:54:23,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 14:54:23,701 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8beb3d080ed1cdf928164eb126128b28 in 912ms, sequenceid=304, compaction requested=true 2023-04-22 14:54:23,701 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:23,701 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:23,703 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@962f334[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:23,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:23,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,708 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:23,708 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,708 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:23,709 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:23,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:23,712 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:23,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:23,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:23,712 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:23,712 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:23,714 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,714 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,714 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,714 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,722 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:23,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:23,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:23,723 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:23,723 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:23,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:23,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:23,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 14:54:23,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:23,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:23,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:23,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:23,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:23,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:23,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175323773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175323773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175323774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175323775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,797 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422e2996570664343eb85bc67e0b4d9837d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2996570664343eb85bc67e0b4d9837d 2023-04-22 14:54:23,823 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:23,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:23,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:23,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:23,826 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:23,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:23,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:23,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:23,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,828 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:23,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:23,828 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:23,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:23,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:23,832 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:23,834 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96d4188[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:23,834 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,835 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,835 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:23,837 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:23,839 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:23,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:23,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:23,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:23,840 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:23,841 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,841 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:23,846 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:23,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,846 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:23,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,846 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:23,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:23,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175323877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175323877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175323878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175323885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,947 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:23,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:23,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:23,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:23,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:23,950 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:23,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:23,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:23,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:23,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:23,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:23,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:23,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:23,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:23,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:23,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:23,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:23,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:23,957 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:23,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472b1f0f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:23,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:23,961 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,961 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:23,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:23,966 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:23,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:23,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:23,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:23,966 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:23,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:23,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:23,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:23,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:23,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:23,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:23,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:23,980 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:23,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:23,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:23,980 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:23,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,080 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175324080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175324080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175324082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,084 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,085 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,087 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,087 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,087 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,087 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,089 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175324087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24296763[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,099 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,101 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,101 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,108 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,108 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,109 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,208 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/dec76da16054437780207985981fa3e8 2023-04-22 14:54:24,208 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,213 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,214 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,214 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,217 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,221 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397b4a59[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,227 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,229 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,229 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,232 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,236 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,236 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,236 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,336 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,340 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157bd503[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,348 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,348 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,350 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,352 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,354 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,355 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,356 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,356 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,356 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,357 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,360 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,361 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175324383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175324384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175324385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175324393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,461 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,465 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,465 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,468 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,468 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,468 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,468 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,473 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,474 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3763c4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,479 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,479 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,481 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,481 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,481 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,486 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,486 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,486 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,487 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,487 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,587 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,590 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,595 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,595 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,595 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,595 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,597 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ca1f9e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,598 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,598 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,602 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,604 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,606 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,606 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,608 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,608 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,608 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,612 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,612 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,612 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/551da50c238147938c45f8f200d3db51 2023-04-22 14:54:24,713 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,716 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,719 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,720 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,720 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,722 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,722 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,723 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,725 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680b8fc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,725 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,726 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,726 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,726 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,727 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,727 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,727 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,730 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,730 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,730 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,730 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,730 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,732 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,733 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,734 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,736 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,736 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,736 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,737 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,737 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,737 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,837 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,841 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,845 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,845 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,845 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,845 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d98a6e2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,849 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,849 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,851 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,853 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,856 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,856 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,857 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,858 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,861 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,862 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,862 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175324884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175324886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175324887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175324898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,962 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:24,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:24,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:24,965 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:24,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:24,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:24,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:24,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:24,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,977 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:24,978 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:24,978 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:24,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c80d5d6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:24,982 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:24,982 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,983 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:24,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:24,985 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:24,987 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:24,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:24,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:24,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:24,987 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:24,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:24,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:24,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:24,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:24,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:24,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:24,999 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:24,999 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:24,999 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:25,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/56886a7daa234fe1a3b5945feb531ee8 2023-04-22 14:54:25,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/dec76da16054437780207985981fa3e8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8 2023-04-22 14:54:25,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8, entries=250, sequenceid=322, filesize=39.2 K 2023-04-22 14:54:25,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/551da50c238147938c45f8f200d3db51 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51 2023-04-22 14:54:25,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51, entries=150, sequenceid=322, filesize=11.9 K 2023-04-22 14:54:25,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/56886a7daa234fe1a3b5945feb531ee8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8 2023-04-22 14:54:25,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8, entries=150, sequenceid=322, filesize=11.9 K 2023-04-22 14:54:25,060 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8beb3d080ed1cdf928164eb126128b28 in 1299ms, sequenceid=322, compaction requested=true 2023-04-22 14:54:25,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:25,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:25,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:25,060 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:25,060 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:25,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:25,062 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:25,062 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144506 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:25,062 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/B is initiating minor compaction (all files) 2023-04-22 14:54:25,062 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/A is initiating minor compaction (all files) 2023-04-22 14:54:25,062 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/B in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:25,063 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/A in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:25,063 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/c76be6e2e9b84ee7bef8e5f38d1bb88e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=59.7 K 2023-04-22 14:54:25,063 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/3c6bd4f882b94932b610cc9a4da56ef2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=141.1 K 2023-04-22 14:54:25,063 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c76be6e2e9b84ee7bef8e5f38d1bb88e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259135 2023-04-22 14:54:25,063 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3c6bd4f882b94932b610cc9a4da56ef2, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259135 2023-04-22 14:54:25,063 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2608c22f0346468caa8c11e5d5046f7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682175259759 2023-04-22 14:54:25,064 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ded955c0200a446280e575ec9b7717a7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682175259759 2023-04-22 14:54:25,064 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5494a89152c74019b9f49dd54f163658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682175260393 2023-04-22 14:54:25,064 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c61cea526f2e44169d783ad7dcce859d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682175260393 2023-04-22 14:54:25,065 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 81e2d382051044468d02a27df9298fa1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175261520 2023-04-22 14:54:25,065 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d2518eeb3d6c443486dca8d71492b8d1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175261520 2023-04-22 14:54:25,065 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 551da50c238147938c45f8f200d3db51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682175263653 2023-04-22 14:54:25,065 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting dec76da16054437780207985981fa3e8, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682175263653 2023-04-22 14:54:25,078 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#B#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:25,082 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#A#compaction#151 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:25,090 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/3ab9b11b572c4d35a48761e0f455b675 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/3ab9b11b572c4d35a48761e0f455b675 2023-04-22 14:54:25,096 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/B of 8beb3d080ed1cdf928164eb126128b28 into 3ab9b11b572c4d35a48761e0f455b675(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:25,096 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:25,096 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/B, priority=11, startTime=1682175265060; duration=0sec 2023-04-22 14:54:25,096 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:25,096 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:25,098 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:25,098 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/C is initiating minor compaction (all files) 2023-04-22 14:54:25,098 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/C in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:25,098 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4878f7e08c8e4bdba3c6481b9ccbb699, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=59.7 K 2023-04-22 14:54:25,098 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4878f7e08c8e4bdba3c6481b9ccbb699, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175259135 2023-04-22 14:54:25,099 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8d2bd2f9c2a948138d203d4ef322ac58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682175259759 2023-04-22 14:54:25,099 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c4a63ce332fd49fab3ef5884abcd24f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682175260393 2023-04-22 14:54:25,099 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 34a9a5c3b37c46d593a799e16feabb1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175261520 2023-04-22 14:54:25,099 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:25,100 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 56886a7daa234fe1a3b5945feb531ee8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682175263653 2023-04-22 14:54:25,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:25,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:25,103 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:25,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:25,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:25,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:25,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:25,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:25,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:25,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:25,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:25,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:25,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:25,109 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:25,109 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:25,111 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#C#compaction#152 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:25,125 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/037ee9e9c7774d71bd8847bcc727a940 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/037ee9e9c7774d71bd8847bcc727a940 2023-04-22 14:54:25,131 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/C of 8beb3d080ed1cdf928164eb126128b28 into 037ee9e9c7774d71bd8847bcc727a940(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:25,131 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:25,131 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/C, priority=11, startTime=1682175265060; duration=0sec 2023-04-22 14:54:25,132 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:25,134 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304228f14cd935a6b4a6b953f9f1b1bc4100b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f14cd935a6b4a6b953f9f1b1bc4100b 2023-04-22 14:54:25,140 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/4dbd361abd0644dab3005e8c87ca8419 2023-04-22 14:54:25,551 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/29eacc95b2af4f9987807427b3474d53 2023-04-22 14:54:25,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:25,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:25,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:25,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175325907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:25,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175325908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:25,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175325909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:25,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56372 deadline: 1682175325910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,964 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/ac7c2dcc46974fadb81e28f984a3f722 2023-04-22 14:54:25,969 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/4dbd361abd0644dab3005e8c87ca8419 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4dbd361abd0644dab3005e8c87ca8419 2023-04-22 14:54:25,975 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4dbd361abd0644dab3005e8c87ca8419, entries=150, sequenceid=337, filesize=25.4 K 2023-04-22 14:54:25,976 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/29eacc95b2af4f9987807427b3474d53 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53 2023-04-22 14:54:25,981 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53, entries=150, sequenceid=337, filesize=11.9 K 2023-04-22 14:54:25,982 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/ac7c2dcc46974fadb81e28f984a3f722 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722 2023-04-22 14:54:25,987 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722, entries=150, sequenceid=337, filesize=11.9 K 2023-04-22 14:54:25,988 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8beb3d080ed1cdf928164eb126128b28 in 880ms, sequenceid=337, compaction requested=false 2023-04-22 14:54:25,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:25,988 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:25,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49831337[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:25,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:25,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:25,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:25,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:25,999 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:25,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:25,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:25,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:25,999 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:26,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:26,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:26,002 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:26,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:26,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:26,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:26,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:26,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:26,004 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:26,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,004 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:26,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:26,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:26,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:26,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:26,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,028 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c193d1681c3d4dfcbd9b40c35156a72a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c193d1681c3d4dfcbd9b40c35156a72a 2023-04-22 14:54:26,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175326023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175326029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175326030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,045 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/13d32948aa21439e8bd90dd595613cdb 2023-04-22 14:54:26,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/9db841d4723e40f2ad12013e9819b62b 2023-04-22 14:54:26,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/7440279bc41b4ff29ef51507810b255f 2023-04-22 14:54:26,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/13d32948aa21439e8bd90dd595613cdb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/13d32948aa21439e8bd90dd595613cdb 2023-04-22 14:54:26,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/13d32948aa21439e8bd90dd595613cdb, entries=250, sequenceid=358, filesize=39.2 K 2023-04-22 14:54:26,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/9db841d4723e40f2ad12013e9819b62b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b 2023-04-22 14:54:26,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b, entries=150, sequenceid=358, filesize=11.9 K 2023-04-22 14:54:26,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/7440279bc41b4ff29ef51507810b255f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f 2023-04-22 14:54:26,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f, entries=150, sequenceid=358, filesize=11.9 K 2023-04-22 14:54:26,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8beb3d080ed1cdf928164eb126128b28 in 77ms, sequenceid=358, compaction requested=true 2023-04-22 14:54:26,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:26,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:26,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:26,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 14:54:26,091 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-04-22 14:54:26,092 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 14:54:26,092 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 14:54:26,092 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. because compaction request was cancelled 2023-04-22 14:54:26,092 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:26,093 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:26,094 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/B is initiating minor compaction (all files) 2023-04-22 14:54:26,094 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/B in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:26,094 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/3ab9b11b572c4d35a48761e0f455b675, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=36.2 K 2023-04-22 14:54:26,095 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3ab9b11b572c4d35a48761e0f455b675, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682175263653 2023-04-22 14:54:26,096 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 29eacc95b2af4f9987807427b3474d53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682175263772 2023-04-22 14:54:26,096 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9db841d4723e40f2ad12013e9819b62b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682175265906 2023-04-22 14:54:26,101 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#B#compaction#159 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:26,105 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:26,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:26,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:26,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:26,108 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:26,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:26,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:26,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:26,108 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:26,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,111 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:26,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:26,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:26,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:26,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:26,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:26,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:26,113 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:26,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:26,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:26,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175326151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175326154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175326155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175326255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175326258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175326258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,310 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/e38060e3fb4349e285d4bfbeb0bf02dd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/e38060e3fb4349e285d4bfbeb0bf02dd 2023-04-22 14:54:26,315 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/A of 8beb3d080ed1cdf928164eb126128b28 into e38060e3fb4349e285d4bfbeb0bf02dd(size=25.9 K), total size for store is 90.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-22 14:54:26,315 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:26,315 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/A, priority=11, startTime=1682175265060; duration=1sec 2023-04-22 14:54:26,315 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:26,315 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:26,318 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:26,318 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 8beb3d080ed1cdf928164eb126128b28/C is initiating minor compaction (all files) 2023-04-22 14:54:26,319 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8beb3d080ed1cdf928164eb126128b28/C in TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:26,319 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/037ee9e9c7774d71bd8847bcc727a940, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp, totalSize=36.2 K 2023-04-22 14:54:26,320 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 037ee9e9c7774d71bd8847bcc727a940, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682175263653 2023-04-22 14:54:26,320 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ac7c2dcc46974fadb81e28f984a3f722, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682175263772 2023-04-22 14:54:26,321 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7440279bc41b4ff29ef51507810b255f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682175265906 2023-04-22 14:54:26,327 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8beb3d080ed1cdf928164eb126128b28#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:26,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175326474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175326474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175326474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,511 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/1070ed3706d54c26822285d454becdb0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1070ed3706d54c26822285d454becdb0 2023-04-22 14:54:26,516 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/B of 8beb3d080ed1cdf928164eb126128b28 into 1070ed3706d54c26822285d454becdb0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:26,516 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:26,516 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/B, priority=13, startTime=1682175266091; duration=0sec 2023-04-22 14:54:26,516 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:26,529 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304228a9b452d37be4622b6d8971226a0db78 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228a9b452d37be4622b6d8971226a0db78 2023-04-22 14:54:26,738 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/b5d1afae173840dab22ce07df5ce956d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/b5d1afae173840dab22ce07df5ce956d 2023-04-22 14:54:26,743 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8beb3d080ed1cdf928164eb126128b28/C of 8beb3d080ed1cdf928164eb126128b28 into b5d1afae173840dab22ce07df5ce956d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:26,743 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:26,743 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., storeName=8beb3d080ed1cdf928164eb126128b28/C, priority=13, startTime=1682175266091; duration=0sec 2023-04-22 14:54:26,743 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:26,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175326778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175326778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:26,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175326780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:26,937 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/efa59e25f48a4eccb011a55dd7d25ae7 2023-04-22 14:54:27,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175327282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175327284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:27,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175327285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,350 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/1d04da62a60744e2856f31705ecfe87d 2023-04-22 14:54:27,761 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/9b2b97309cb14dd9a716c38f854b0b63 2023-04-22 14:54:27,767 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/efa59e25f48a4eccb011a55dd7d25ae7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/efa59e25f48a4eccb011a55dd7d25ae7 2023-04-22 14:54:27,770 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/efa59e25f48a4eccb011a55dd7d25ae7, entries=150, sequenceid=370, filesize=25.4 K 2023-04-22 14:54:27,771 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/1d04da62a60744e2856f31705ecfe87d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1d04da62a60744e2856f31705ecfe87d 2023-04-22 14:54:27,784 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1d04da62a60744e2856f31705ecfe87d, entries=150, sequenceid=370, filesize=11.9 K 2023-04-22 14:54:27,790 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/9b2b97309cb14dd9a716c38f854b0b63 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/9b2b97309cb14dd9a716c38f854b0b63 2023-04-22 14:54:27,795 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/9b2b97309cb14dd9a716c38f854b0b63, entries=150, sequenceid=370, filesize=11.9 K 2023-04-22 14:54:27,796 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8beb3d080ed1cdf928164eb126128b28 in 1683ms, sequenceid=370, compaction requested=true 2023-04-22 14:54:27,796 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:27,796 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:27,798 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:27,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac715a3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:27,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:27,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,801 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:27,801 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,802 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:27,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:27,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:27,806 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:27,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:27,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:27,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:27,806 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:27,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:27,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,809 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:27,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:27,813 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:27,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:27,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:27,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:27,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:27,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:27,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,814 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:27,814 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:27,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:27,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:27,914 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:27,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:27,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:27,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:27,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:27,917 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:27,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:27,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:27,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:27,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:27,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:27,919 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:27,919 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:27,919 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:27,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:27,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:27,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:27,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:27,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:27,921 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:27,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:27,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:27,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:27,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:27,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:27,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:27,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:27,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:27,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175327941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175328044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175328247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175328291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175328292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175328296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,336 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422397642ff7ece4f8a9efeee7298641652 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422397642ff7ece4f8a9efeee7298641652 2023-04-22 14:54:28,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:28,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175328550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:28,742 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=393, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/0588e0192d244438b18a124ee16bb724 2023-04-22 14:54:29,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175329056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,156 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/ffcb6a086c1c4823a258bee57f53e5c3 2023-04-22 14:54:29,567 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/3452e548df88463e95d495ace73f712e 2023-04-22 14:54:29,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/0588e0192d244438b18a124ee16bb724 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/0588e0192d244438b18a124ee16bb724 2023-04-22 14:54:29,576 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/0588e0192d244438b18a124ee16bb724, entries=150, sequenceid=393, filesize=25.4 K 2023-04-22 14:54:29,577 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/ffcb6a086c1c4823a258bee57f53e5c3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ffcb6a086c1c4823a258bee57f53e5c3 2023-04-22 14:54:29,581 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ffcb6a086c1c4823a258bee57f53e5c3, entries=150, sequenceid=393, filesize=11.9 K 2023-04-22 14:54:29,582 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/3452e548df88463e95d495ace73f712e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3452e548df88463e95d495ace73f712e 2023-04-22 14:54:29,586 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3452e548df88463e95d495ace73f712e, entries=150, sequenceid=393, filesize=11.9 K 2023-04-22 14:54:29,587 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8beb3d080ed1cdf928164eb126128b28 in 1666ms, sequenceid=393, compaction requested=true 2023-04-22 14:54:29,587 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:29,587 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:29,587 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:29,587 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:29,587 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,587 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:29,587 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:29,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:29,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426015af[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:29,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:29,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,596 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:29,596 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,596 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:29,597 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:29,600 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:29,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:29,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:29,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:29,600 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:29,602 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,602 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:29,602 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:29,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,610 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:29,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:29,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:29,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:29,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:29,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:29,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,611 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:29,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:29,611 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:29,711 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:29,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:29,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:29,715 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:29,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:29,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:29,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:29,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:29,717 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:29,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:29,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:29,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:29,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:29,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:29,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:29,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:29,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:29,719 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:29,719 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:29,734 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220f85661a2a144e0eb698308d7d25eb90 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220f85661a2a144e0eb698308d7d25eb90 2023-04-22 14:54:29,741 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/82af108395c6419eba25be13bdd6ced8 2023-04-22 14:54:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:30,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:30,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175330113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,152 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/838e267386bc46eaa22e53dd3af2f42b 2023-04-22 14:54:30,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175330216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56338 deadline: 1682175330298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,302 DEBUG [Thread-825] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:30,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56324 deadline: 1682175330305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,309 DEBUG [Thread-821] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:30,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56366 deadline: 1682175330309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,312 DEBUG [Thread-817] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:30,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175330419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,565 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/c59058a5889a49888b29d19036879928 2023-04-22 14:54:30,570 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/82af108395c6419eba25be13bdd6ced8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/82af108395c6419eba25be13bdd6ced8 2023-04-22 14:54:30,574 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/82af108395c6419eba25be13bdd6ced8, entries=150, sequenceid=405, filesize=25.4 K 2023-04-22 14:54:30,575 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/838e267386bc46eaa22e53dd3af2f42b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/838e267386bc46eaa22e53dd3af2f42b 2023-04-22 14:54:30,579 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/838e267386bc46eaa22e53dd3af2f42b, entries=150, sequenceid=405, filesize=11.9 K 2023-04-22 14:54:30,581 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/c59058a5889a49888b29d19036879928 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c59058a5889a49888b29d19036879928 2023-04-22 14:54:30,590 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c59058a5889a49888b29d19036879928, entries=150, sequenceid=405, filesize=11.9 K 2023-04-22 14:54:30,591 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8beb3d080ed1cdf928164eb126128b28 in 872ms, sequenceid=405, compaction requested=true 2023-04-22 14:54:30,591 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:30,591 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:30,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:30,594 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,594 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b44650[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:30,596 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:30,596 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,597 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:30,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:30,597 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:30,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:30,600 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,600 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:30,602 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:30,602 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:30,602 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:30,602 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:30,602 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:30,604 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,604 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:30,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,604 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:30,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:30,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:30,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:30,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:30,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:30,609 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:30,609 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:30,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:30,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:30,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:30,709 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:30,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:30,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:30,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:30,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:30,713 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:30,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:30,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:30,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:30,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:30,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:30,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:30,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:30,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:30,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:30,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:30,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:30,717 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:30,717 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:30,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:30,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:30,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175330754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175330857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:31,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175331059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,135 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ae8c6adff8734fea927ebe93ac22acbe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ae8c6adff8734fea927ebe93ac22acbe 2023-04-22 14:54:31,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175331363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,541 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/8436100629be4b749d28c02701510a55 2023-04-22 14:54:31,557 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/d1692e0274af421c885dd74b825f0b0e 2023-04-22 14:54:31,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:31,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175331869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,968 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/88d486d822544c6584385e3bd165c3e9 2023-04-22 14:54:31,973 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/8436100629be4b749d28c02701510a55 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/8436100629be4b749d28c02701510a55 2023-04-22 14:54:31,977 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/8436100629be4b749d28c02701510a55, entries=150, sequenceid=425, filesize=25.4 K 2023-04-22 14:54:31,978 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/d1692e0274af421c885dd74b825f0b0e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/d1692e0274af421c885dd74b825f0b0e 2023-04-22 14:54:31,982 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/d1692e0274af421c885dd74b825f0b0e, entries=150, sequenceid=425, filesize=11.9 K 2023-04-22 14:54:31,984 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/88d486d822544c6584385e3bd165c3e9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/88d486d822544c6584385e3bd165c3e9 2023-04-22 14:54:31,988 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/88d486d822544c6584385e3bd165c3e9, entries=150, sequenceid=425, filesize=11.9 K 2023-04-22 14:54:31,989 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8beb3d080ed1cdf928164eb126128b28 in 1272ms, sequenceid=425, compaction requested=true 2023-04-22 14:54:31,989 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:31,989 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:31,992 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:31,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:31,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697ff0da[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:31,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:31,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:31,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:31,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:31,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:31,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:31,999 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:32,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:32,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:32,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:32,000 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:32,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:32,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:32,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:32,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:32,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:32,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:32,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:32,010 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:32,010 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:32,010 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:32,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:32,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,111 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:32,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:32,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:32,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:32,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:32,115 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:32,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:32,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:32,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:32,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:32,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:32,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:32,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:32,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:32,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:32,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:32,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:32,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:32,118 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:32,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:32,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:32,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:32,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:32,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175332334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:32,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175332437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,536 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304229c096a8f319c42b281cdd310d41d4576 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229c096a8f319c42b281cdd310d41d4576 2023-04-22 14:54:32,542 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=437, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/4850468e4bfe4312bf356b0c89a14f52 2023-04-22 14:54:32,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175332641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56372 deadline: 1682175332880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1682175332948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:32,968 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/7e955a2c2afd46a0a3e89eb2734fafc7 2023-04-22 14:54:32,986 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/5685d6375e614d4a8ab6ce711296fba4 2023-04-22 14:54:32,994 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/4850468e4bfe4312bf356b0c89a14f52 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4850468e4bfe4312bf356b0c89a14f52 2023-04-22 14:54:32,999 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4850468e4bfe4312bf356b0c89a14f52, entries=150, sequenceid=437, filesize=25.4 K 2023-04-22 14:54:33,000 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/7e955a2c2afd46a0a3e89eb2734fafc7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7e955a2c2afd46a0a3e89eb2734fafc7 2023-04-22 14:54:33,004 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7e955a2c2afd46a0a3e89eb2734fafc7, entries=150, sequenceid=437, filesize=11.9 K 2023-04-22 14:54:33,005 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/5685d6375e614d4a8ab6ce711296fba4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5685d6375e614d4a8ab6ce711296fba4 2023-04-22 14:54:33,009 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5685d6375e614d4a8ab6ce711296fba4, entries=150, sequenceid=437, filesize=11.9 K 2023-04-22 14:54:33,010 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8beb3d080ed1cdf928164eb126128b28 in 892ms, sequenceid=437, compaction requested=true 2023-04-22 14:54:33,010 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:33,010 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:33,012 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:33,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1be78a5b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:33,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:33,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,015 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:33,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:33,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:33,019 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:33,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:33,022 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:33,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:33,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:33,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:33,023 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:33,024 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,024 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,024 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:33,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,026 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:33,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:33,031 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:33,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:33,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:33,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:33,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,032 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,032 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,032 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:33,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:33,032 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:33,132 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:33,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:33,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:33,136 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:33,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:33,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:33,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:33,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,138 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,138 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:33,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:33,138 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,138 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:33,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:33,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:33,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:33,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. started... 2023-04-22 14:54:33,140 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:33,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:33,165 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422709e150cbe8b4ef48b01d50f0f7bdf5f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422709e150cbe8b4ef48b01d50f0f7bdf5f 2023-04-22 14:54:33,174 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=457, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/420659dbb3fd4f959da1d1c0e1cf5032 2023-04-22 14:54:33,190 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/b4cac35212314d7aa2d7b1b0bd69eafc 2023-04-22 14:54:33,225 DEBUG [Thread-830] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:51853 2023-04-22 14:54:33,225 DEBUG [Thread-830] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:33,227 DEBUG [Thread-828] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:51853 2023-04-22 14:54:33,227 DEBUG [Thread-828] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:33,229 DEBUG [Thread-832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:51853 2023-04-22 14:54:33,229 DEBUG [Thread-832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:33,231 DEBUG [Thread-834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:51853 2023-04-22 14:54:33,231 DEBUG [Thread-834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:33,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:33,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. as already flushing 2023-04-22 14:54:33,453 DEBUG [Thread-823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:51853 2023-04-22 14:54:33,453 DEBUG [Thread-823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:33,603 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/f5f02a1c0ae34f10842e33ec082dcd37 2023-04-22 14:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/420659dbb3fd4f959da1d1c0e1cf5032 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/420659dbb3fd4f959da1d1c0e1cf5032 2023-04-22 14:54:33,612 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/420659dbb3fd4f959da1d1c0e1cf5032, entries=150, sequenceid=457, filesize=25.4 K 2023-04-22 14:54:33,613 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/b4cac35212314d7aa2d7b1b0bd69eafc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b4cac35212314d7aa2d7b1b0bd69eafc 2023-04-22 14:54:33,616 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b4cac35212314d7aa2d7b1b0bd69eafc, entries=150, sequenceid=457, filesize=11.9 K 2023-04-22 14:54:33,617 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/f5f02a1c0ae34f10842e33ec082dcd37 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/f5f02a1c0ae34f10842e33ec082dcd37 2023-04-22 14:54:33,620 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/f5f02a1c0ae34f10842e33ec082dcd37, entries=150, sequenceid=457, filesize=11.9 K 2023-04-22 14:54:33,621 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=6.71 KB/6870 for 8beb3d080ed1cdf928164eb126128b28 in 481ms, sequenceid=457, compaction requested=true 2023-04-22 14:54:33,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:33,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:33,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:33,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:33,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:33,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,623 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:33,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e829e5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:33,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:33,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,626 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:33,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:33,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:33,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:33,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:33,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:33,631 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:33,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:33,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:33,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:33,632 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:33,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:33,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:33,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:33,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:33,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:33,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:33,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:33,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:33,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:33,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:33,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:33,637 DEBUG [Thread-827] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:33,637 DEBUG [Thread-827] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:33,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:33,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:33,737 DEBUG [Thread-827] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:34,325 DEBUG [Thread-825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:51853 2023-04-22 14:54:34,326 DEBUG [Thread-825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:34,336 DEBUG [Thread-821] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:51853 2023-04-22 14:54:34,336 DEBUG [Thread-821] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:34,338 DEBUG [Thread-817] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:51853 2023-04-22 14:54:34,338 DEBUG [Thread-817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:34,897 DEBUG [Thread-819] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:51853 2023-04-22 14:54:34,897 DEBUG [Thread-819] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:34,897 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:54:34,897 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-04-22 14:54:34,897 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-22 14:54:34,897 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5532 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5351 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 2277 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 6831 rows 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 2293 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 6867 rows 2023-04-22 14:54:34,898 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:54:34,898 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:51853 2023-04-22 14:54:34,898 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:34,899 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:54:34,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:54:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:34,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:34,905 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175274905"}]},"ts":"1682175274905"} 2023-04-22 14:54:34,906 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:54:34,912 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:54:34,913 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, UNASSIGN}] 2023-04-22 14:54:34,914 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, UNASSIGN 2023-04-22 14:54:34,915 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:34,915 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175274915"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175274915"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175274915"}]},"ts":"1682175274915"} 2023-04-22 14:54:34,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:35,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:35,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:35,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8beb3d080ed1cdf928164eb126128b28, disabling compactions & flushes 2023-04-22 14:54:35,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:35,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:35,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. after waiting 0 ms 2023-04-22 14:54:35,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:35,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8beb3d080ed1cdf928164eb126128b28 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=A 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=B 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8beb3d080ed1cdf928164eb126128b28, store=C 2023-04-22 14:54:35,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:35,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:35,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422090a6d116ec34d1ba0f4b9a33a32cd44 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422090a6d116ec34d1ba0f4b9a33a32cd44 2023-04-22 14:54:35,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:35,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/5471107c2b63468080195166cb2bab30 2023-04-22 14:54:36,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:36,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/a1d3680f613a4ae7a6f0a04dc643679c 2023-04-22 14:54:36,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/a94f793020e949eab468c408eb172215 2023-04-22 14:54:36,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/A/5471107c2b63468080195166cb2bab30 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5471107c2b63468080195166cb2bab30 2023-04-22 14:54:36,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5471107c2b63468080195166cb2bab30, entries=100, sequenceid=465, filesize=18.6 K 2023-04-22 14:54:36,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/B/a1d3680f613a4ae7a6f0a04dc643679c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a1d3680f613a4ae7a6f0a04dc643679c 2023-04-22 14:54:36,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a1d3680f613a4ae7a6f0a04dc643679c, entries=100, sequenceid=465, filesize=9.5 K 2023-04-22 14:54:36,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/.tmp/C/a94f793020e949eab468c408eb172215 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a94f793020e949eab468c408eb172215 2023-04-22 14:54:36,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a94f793020e949eab468c408eb172215, entries=100, sequenceid=465, filesize=9.5 K 2023-04-22 14:54:36,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 8beb3d080ed1cdf928164eb126128b28 in 1656ms, sequenceid=465, compaction requested=true 2023-04-22 14:54:36,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/738a10c84b8a429699a527daa408ca58, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ce39317b9382475a9affec9dffab9953, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/3c6bd4f882b94932b610cc9a4da56ef2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8] to archive 2023-04-22 14:54:36,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:36,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ee574aa943e14586a15b83cf180875f3 2023-04-22 14:54:36,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/1c147581d2544d4790bb90c54f63d30f 2023-04-22 14:54:36,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/08f9463c49c54ec48cf38d8ca04a9e65 2023-04-22 14:54:36,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/f7fe989d1a984447beb2cef53577899e 2023-04-22 14:54:36,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/738a10c84b8a429699a527daa408ca58 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/738a10c84b8a429699a527daa408ca58 2023-04-22 14:54:36,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5db98feed5504386a1571c962bd62d3c 2023-04-22 14:54:36,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/06acd2d4b02849268a4ec8316df97d07 2023-04-22 14:54:36,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ce39317b9382475a9affec9dffab9953 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ce39317b9382475a9affec9dffab9953 2023-04-22 14:54:36,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/763047f0e05f459595232ba4a12b71bc 2023-04-22 14:54:36,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/42093676e58243f0a52559020f56d709 2023-04-22 14:54:36,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/3c6bd4f882b94932b610cc9a4da56ef2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/3c6bd4f882b94932b610cc9a4da56ef2 2023-04-22 14:54:36,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/ded955c0200a446280e575ec9b7717a7 2023-04-22 14:54:36,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/c61cea526f2e44169d783ad7dcce859d 2023-04-22 14:54:36,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/d2518eeb3d6c443486dca8d71492b8d1 2023-04-22 14:54:36,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/dec76da16054437780207985981fa3e8 2023-04-22 14:54:36,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b7cf5bf5598b43c9b41abef6bb6ed66f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/84c83974b60f439bbc185e22c3cb13df, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/c76be6e2e9b84ee7bef8e5f38d1bb88e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/3ab9b11b572c4d35a48761e0f455b675, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b] to archive 2023-04-22 14:54:36,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:36,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/72de8988de754cf6a13657fe4c0a00be 2023-04-22 14:54:36,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/e480580b39f648acaf110918f745f988 2023-04-22 14:54:36,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/12e5873a95854835852cced33ed47d2a 2023-04-22 14:54:36,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b7cf5bf5598b43c9b41abef6bb6ed66f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b7cf5bf5598b43c9b41abef6bb6ed66f 2023-04-22 14:54:36,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a7d5123958b8408eab28990704e079a7 2023-04-22 14:54:36,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7ced2f25e6e74c3894930795b758a343 2023-04-22 14:54:36,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/84c83974b60f439bbc185e22c3cb13df to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/84c83974b60f439bbc185e22c3cb13df 2023-04-22 14:54:36,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ba7764ac30eb4de48733f119d36c891c 2023-04-22 14:54:36,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/cad98cf02667458396f76736ddb3bdc8 2023-04-22 14:54:36,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/c76be6e2e9b84ee7bef8e5f38d1bb88e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/c76be6e2e9b84ee7bef8e5f38d1bb88e 2023-04-22 14:54:36,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1bdf7dab4449488cbc0e6f7bb7ae45ec 2023-04-22 14:54:36,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/2608c22f0346468caa8c11e5d5046f7e 2023-04-22 14:54:36,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/5494a89152c74019b9f49dd54f163658 2023-04-22 14:54:36,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/81e2d382051044468d02a27df9298fa1 2023-04-22 14:54:36,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/3ab9b11b572c4d35a48761e0f455b675 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/3ab9b11b572c4d35a48761e0f455b675 2023-04-22 14:54:36,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/551da50c238147938c45f8f200d3db51 2023-04-22 14:54:36,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/29eacc95b2af4f9987807427b3474d53 2023-04-22 14:54:36,787 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/9db841d4723e40f2ad12013e9819b62b 2023-04-22 14:54:36,790 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/60de038d47d34900b4433791d1a1c2bd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/6984e151c5c747d880a3e0fe330ccbba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4878f7e08c8e4bdba3c6481b9ccbb699, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/037ee9e9c7774d71bd8847bcc727a940, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f] to archive 2023-04-22 14:54:36,790 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:36,791 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/2364cfe6ac484910b77a75871e4a3824 2023-04-22 14:54:36,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/e3908b47225e4d8c848de96d7c55a8fb 2023-04-22 14:54:36,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82cf5011c266447995e19c286dfa6caa 2023-04-22 14:54:36,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/60de038d47d34900b4433791d1a1c2bd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/60de038d47d34900b4433791d1a1c2bd 2023-04-22 14:54:36,795 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/82107e05b2fb4aec81ee2bc26828d71a 2023-04-22 14:54:36,796 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5c0c39b2574f4abd80040362d1516178 2023-04-22 14:54:36,797 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/6984e151c5c747d880a3e0fe330ccbba to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/6984e151c5c747d880a3e0fe330ccbba 2023-04-22 14:54:36,798 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3b1ddfa3b14e40eba7cb9b7be8daa1e0 2023-04-22 14:54:36,799 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4aa96dd89a7c4bd192fb308ed7070942 2023-04-22 14:54:36,800 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4878f7e08c8e4bdba3c6481b9ccbb699 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/4878f7e08c8e4bdba3c6481b9ccbb699 2023-04-22 14:54:36,801 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a0407cb72fdb43d794345ba2751f4a5f 2023-04-22 14:54:36,802 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/8d2bd2f9c2a948138d203d4ef322ac58 2023-04-22 14:54:36,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c4a63ce332fd49fab3ef5884abcd24f4 2023-04-22 14:54:36,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/34a9a5c3b37c46d593a799e16feabb1c 2023-04-22 14:54:36,805 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/037ee9e9c7774d71bd8847bcc727a940 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/037ee9e9c7774d71bd8847bcc727a940 2023-04-22 14:54:36,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/56886a7daa234fe1a3b5945feb531ee8 2023-04-22 14:54:36,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/ac7c2dcc46974fadb81e28f984a3f722 2023-04-22 14:54:36,808 DEBUG [StoreCloser-TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/7440279bc41b4ff29ef51507810b255f 2023-04-22 14:54:36,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits/468.seqid, newMaxSeqId=468, maxSeqId=4 2023-04-22 14:54:36,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28. 2023-04-22 14:54:36,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8beb3d080ed1cdf928164eb126128b28: 2023-04-22 14:54:36,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:36,816 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8beb3d080ed1cdf928164eb126128b28, regionState=CLOSED 2023-04-22 14:54:36,816 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175276816"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175276816"}]},"ts":"1682175276816"} 2023-04-22 14:54:36,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-22 14:54:36,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 8beb3d080ed1cdf928164eb126128b28, server=jenkins-hbase4.apache.org,32935,1682175218140 in 1.9010 sec 2023-04-22 14:54:36,819 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-22 14:54:36,819 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8beb3d080ed1cdf928164eb126128b28, UNASSIGN in 1.9050 sec 2023-04-22 14:54:36,820 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175276820"}]},"ts":"1682175276820"} 2023-04-22 14:54:36,821 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:54:36,822 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:54:36,824 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9230 sec 2023-04-22 14:54:37,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:54:37,010 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-22 14:54:37,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:54:37,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,013 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-22 14:54:37,014 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,019 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:37,021 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits] 2023-04-22 14:54:37,024 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/0588e0192d244438b18a124ee16bb724 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/0588e0192d244438b18a124ee16bb724 2023-04-22 14:54:37,025 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/13d32948aa21439e8bd90dd595613cdb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/13d32948aa21439e8bd90dd595613cdb 2023-04-22 14:54:37,026 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/420659dbb3fd4f959da1d1c0e1cf5032 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/420659dbb3fd4f959da1d1c0e1cf5032 2023-04-22 14:54:37,027 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4850468e4bfe4312bf356b0c89a14f52 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4850468e4bfe4312bf356b0c89a14f52 2023-04-22 14:54:37,029 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4dbd361abd0644dab3005e8c87ca8419 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/4dbd361abd0644dab3005e8c87ca8419 2023-04-22 14:54:37,030 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5471107c2b63468080195166cb2bab30 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/5471107c2b63468080195166cb2bab30 2023-04-22 14:54:37,031 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/82af108395c6419eba25be13bdd6ced8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/82af108395c6419eba25be13bdd6ced8 2023-04-22 14:54:37,032 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/8436100629be4b749d28c02701510a55 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/8436100629be4b749d28c02701510a55 2023-04-22 14:54:37,033 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/e38060e3fb4349e285d4bfbeb0bf02dd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/e38060e3fb4349e285d4bfbeb0bf02dd 2023-04-22 14:54:37,034 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/efa59e25f48a4eccb011a55dd7d25ae7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/A/efa59e25f48a4eccb011a55dd7d25ae7 2023-04-22 14:54:37,037 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1070ed3706d54c26822285d454becdb0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1070ed3706d54c26822285d454becdb0 2023-04-22 14:54:37,038 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1d04da62a60744e2856f31705ecfe87d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/1d04da62a60744e2856f31705ecfe87d 2023-04-22 14:54:37,040 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7e955a2c2afd46a0a3e89eb2734fafc7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/7e955a2c2afd46a0a3e89eb2734fafc7 2023-04-22 14:54:37,041 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/838e267386bc46eaa22e53dd3af2f42b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/838e267386bc46eaa22e53dd3af2f42b 2023-04-22 14:54:37,042 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a1d3680f613a4ae7a6f0a04dc643679c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/a1d3680f613a4ae7a6f0a04dc643679c 2023-04-22 14:54:37,044 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b4cac35212314d7aa2d7b1b0bd69eafc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/b4cac35212314d7aa2d7b1b0bd69eafc 2023-04-22 14:54:37,045 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/d1692e0274af421c885dd74b825f0b0e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/d1692e0274af421c885dd74b825f0b0e 2023-04-22 14:54:37,046 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ffcb6a086c1c4823a258bee57f53e5c3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/B/ffcb6a086c1c4823a258bee57f53e5c3 2023-04-22 14:54:37,048 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3452e548df88463e95d495ace73f712e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/3452e548df88463e95d495ace73f712e 2023-04-22 14:54:37,049 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5685d6375e614d4a8ab6ce711296fba4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/5685d6375e614d4a8ab6ce711296fba4 2023-04-22 14:54:37,050 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/88d486d822544c6584385e3bd165c3e9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/88d486d822544c6584385e3bd165c3e9 2023-04-22 14:54:37,051 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/9b2b97309cb14dd9a716c38f854b0b63 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/9b2b97309cb14dd9a716c38f854b0b63 2023-04-22 14:54:37,052 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a94f793020e949eab468c408eb172215 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/a94f793020e949eab468c408eb172215 2023-04-22 14:54:37,053 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/b5d1afae173840dab22ce07df5ce956d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/b5d1afae173840dab22ce07df5ce956d 2023-04-22 14:54:37,054 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c59058a5889a49888b29d19036879928 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/c59058a5889a49888b29d19036879928 2023-04-22 14:54:37,055 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/f5f02a1c0ae34f10842e33ec082dcd37 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/C/f5f02a1c0ae34f10842e33ec082dcd37 2023-04-22 14:54:37,058 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits/468.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28/recovered.edits/468.seqid 2023-04-22 14:54:37,059 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/8beb3d080ed1cdf928164eb126128b28 2023-04-22 14:54:37,059 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:54:37,059 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:54:37,060 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 14:54:37,063 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422090a6d116ec34d1ba0f4b9a33a32cd44 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422090a6d116ec34d1ba0f4b9a33a32cd44 2023-04-22 14:54:37,064 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220f85661a2a144e0eb698308d7d25eb90 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220f85661a2a144e0eb698308d7d25eb90 2023-04-22 14:54:37,065 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221697abf239054c7aab70f7fba592b4c2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221697abf239054c7aab70f7fba592b4c2 2023-04-22 14:54:37,066 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223323b799cbcf45489eaa305b133f8e82 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223323b799cbcf45489eaa305b133f8e82 2023-04-22 14:54:37,067 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223850cb81a9f64ddfb45e0fce7dfae140 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223850cb81a9f64ddfb45e0fce7dfae140 2023-04-22 14:54:37,068 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422397642ff7ece4f8a9efeee7298641652 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422397642ff7ece4f8a9efeee7298641652 2023-04-22 14:54:37,069 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422709e150cbe8b4ef48b01d50f0f7bdf5f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422709e150cbe8b4ef48b01d50f0f7bdf5f 2023-04-22 14:54:37,070 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228a9b452d37be4622b6d8971226a0db78 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228a9b452d37be4622b6d8971226a0db78 2023-04-22 14:54:37,071 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f0bc9e86fbb4e85993d0b9f8f49a94e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f0bc9e86fbb4e85993d0b9f8f49a94e 2023-04-22 14:54:37,072 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f14cd935a6b4a6b953f9f1b1bc4100b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304228f14cd935a6b4a6b953f9f1b1bc4100b 2023-04-22 14:54:37,074 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042298120d386ee44331895115978061e24a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042298120d386ee44331895115978061e24a 2023-04-22 14:54:37,075 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229c096a8f319c42b281cdd310d41d4576 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304229c096a8f319c42b281cdd310d41d4576 2023-04-22 14:54:37,075 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ab947ac2ce49442d8831c9c2d90b4749 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ab947ac2ce49442d8831c9c2d90b4749 2023-04-22 14:54:37,076 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ae8c6adff8734fea927ebe93ac22acbe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ae8c6adff8734fea927ebe93ac22acbe 2023-04-22 14:54:37,077 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b06715bedccc419fb9cc6fdf7bdb2776 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b06715bedccc419fb9cc6fdf7bdb2776 2023-04-22 14:54:37,078 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b5dadb8929b8445e8d1322411b262b1d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b5dadb8929b8445e8d1322411b262b1d 2023-04-22 14:54:37,079 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c193d1681c3d4dfcbd9b40c35156a72a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c193d1681c3d4dfcbd9b40c35156a72a 2023-04-22 14:54:37,081 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c53f8101a98c48dbb880a77afbc0ce32 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c53f8101a98c48dbb880a77afbc0ce32 2023-04-22 14:54:37,082 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d444679f1bc8419a8c15446ff85f5c63 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d444679f1bc8419a8c15446ff85f5c63 2023-04-22 14:54:37,083 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2996570664343eb85bc67e0b4d9837d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2996570664343eb85bc67e0b4d9837d 2023-04-22 14:54:37,084 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ea68a4d5bbab4c0988a91247b29bfcb8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ea68a4d5bbab4c0988a91247b29bfcb8 2023-04-22 14:54:37,084 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:54:37,086 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,088 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:54:37,090 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:54:37,091 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,091 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:54:37,091 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175277091"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:37,092 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:54:37,092 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8beb3d080ed1cdf928164eb126128b28, NAME => 'TestAcidGuarantees,,1682175250466.8beb3d080ed1cdf928164eb126128b28.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:54:37,092 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:54:37,092 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175277092"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:37,094 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:54:37,095 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 85 msec 2023-04-22 14:54:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-22 14:54:37,115 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-22 14:54:37,128 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 234), OpenFileDescriptor=542 (was 546), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 450), ProcessCount=172 (was 172), AvailableMemoryMB=12473 (was 12551) 2023-04-22 14:54:37,140 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=172, AvailableMemoryMB=12472 2023-04-22 14:54:37,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:54:37,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:54:37,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:54:37,146 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:54:37,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-22 14:54:37,146 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:54:37,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:37,147 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:37,148 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 empty. 2023-04-22 14:54:37,148 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:37,148 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:54:37,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:37,557 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:37,558 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5adf52822e4e313e73f3d2c863d10c45, NAME => 'TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:54:37,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:37,964 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:37,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5adf52822e4e313e73f3d2c863d10c45, disabling compactions & flushes 2023-04-22 14:54:37,965 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:37,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:37,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. after waiting 0 ms 2023-04-22 14:54:37,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:37,965 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:37,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:37,967 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:54:37,968 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175277968"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175277968"}]},"ts":"1682175277968"} 2023-04-22 14:54:37,969 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:54:37,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:54:37,970 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175277969"}]},"ts":"1682175277969"} 2023-04-22 14:54:37,970 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:54:37,974 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, ASSIGN}] 2023-04-22 14:54:37,975 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, ASSIGN 2023-04-22 14:54:37,976 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:54:38,127 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5adf52822e4e313e73f3d2c863d10c45, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:38,127 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175278127"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175278127"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175278127"}]},"ts":"1682175278127"} 2023-04-22 14:54:38,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:54:38,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:38,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:38,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5adf52822e4e313e73f3d2c863d10c45, NAME => 'TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:38,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:38,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,286 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,287 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A 2023-04-22 14:54:38,287 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A 2023-04-22 14:54:38,287 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:38,288 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5adf52822e4e313e73f3d2c863d10c45 columnFamilyName A 2023-04-22 14:54:38,288 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(310): Store=5adf52822e4e313e73f3d2c863d10c45/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:38,288 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,290 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B 2023-04-22 14:54:38,290 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B 2023-04-22 14:54:38,290 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:38,290 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5adf52822e4e313e73f3d2c863d10c45 columnFamilyName B 2023-04-22 14:54:38,290 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(310): Store=5adf52822e4e313e73f3d2c863d10c45/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:38,291 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,291 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C 2023-04-22 14:54:38,292 DEBUG [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C 2023-04-22 14:54:38,292 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:54:38,292 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5adf52822e4e313e73f3d2c863d10c45 columnFamilyName C 2023-04-22 14:54:38,292 INFO [StoreOpener-5adf52822e4e313e73f3d2c863d10c45-1] regionserver.HStore(310): Store=5adf52822e4e313e73f3d2c863d10c45/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:38,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:38,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:54:38,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:38,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:54:38,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5adf52822e4e313e73f3d2c863d10c45; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69625670, jitterRate=0.037503331899642944}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:54:38,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:38,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., pid=30, masterSystemTime=1682175278280 2023-04-22 14:54:38,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:38,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:38,301 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5adf52822e4e313e73f3d2c863d10c45, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:38,301 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175278301"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175278301"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175278301"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175278301"}]},"ts":"1682175278301"} 2023-04-22 14:54:38,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-22 14:54:38,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 in 173 msec 2023-04-22 14:54:38,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-22 14:54:38,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, ASSIGN in 329 msec 2023-04-22 14:54:38,305 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:54:38,305 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175278305"}]},"ts":"1682175278305"} 2023-04-22 14:54:38,306 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:54:38,308 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:54:38,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-22 14:54:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-22 14:54:39,251 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-22 14:54:39,251 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,256 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,257 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,259 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,260 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:54:39,262 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:54:39,263 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,267 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,267 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,275 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,276 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,279 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,279 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,283 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,283 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,286 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,287 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,292 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,292 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,300 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,300 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,304 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,305 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,310 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,311 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:54:39,316 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:54:39,320 DEBUG [hconnection-0x4b0a7d16-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:39,327 DEBUG [hconnection-0x2ca0411a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,327 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51682, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,327 DEBUG [hconnection-0x102c2fa9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,328 DEBUG [hconnection-0x24538b64-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,328 DEBUG [hconnection-0x19996b40-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,329 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,329 DEBUG [hconnection-0x749db13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,329 DEBUG [hconnection-0x563ccbc7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,329 DEBUG [hconnection-0x3f6286db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,330 DEBUG [hconnection-0xa1c4839-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,331 DEBUG [hconnection-0xb151086-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:39,331 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,333 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,333 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,333 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:39,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:39,336 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:39,337 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,337 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:39,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:39,339 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,339 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,339 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:39,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,346 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,348 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,348 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:39,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:39,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:39,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:39,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,350 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:39,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:39,357 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,357 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:39,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:39,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:39,358 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:39,358 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:39,358 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:39,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:39,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:39,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:39,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:39,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:39,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:39,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:39,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:39,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:39,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:39,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f4724e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:39,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:39,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,364 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:39,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:39,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:39,371 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:39,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:39,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:39,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:39,371 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:39,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,376 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:39,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:39,382 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:39,382 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:39,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175339376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175339380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175339406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175339406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175339406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,482 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:39,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:39,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:39,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:39,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:39,486 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:39,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:39,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:39,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,489 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,490 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,490 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:39,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:39,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:39,500 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:39,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:39,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:39,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:39,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551ba3ca[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:39,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:39,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:39,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:39,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175339507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175339507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:39,509 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:39,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:39,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:39,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:39,510 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:39,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175339510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175339511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175339511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:39,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,516 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:39,516 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,517 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:39,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,617 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:39,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:39,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:39,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:39,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:39,620 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:39,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:39,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:39,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:39,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:39,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:39,624 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,624 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:39,624 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:39,624 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:39,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:39,626 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:39,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:39,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72894e60[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:39,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,630 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:39,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:39,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:39,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:39,634 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:39,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:39,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:39,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:39,634 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:39,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,638 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:39,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:39,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,655 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:39,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,656 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:39,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175339710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175339710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175339713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175339714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175339714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,756 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:39,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:39,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:39,759 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:39,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:39,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:39,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:39,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:39,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:39,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:39,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:39,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:39,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:39,764 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:39,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:39,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:39,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:39,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,766 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:39,768 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab6e6d5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:39,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:39,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:39,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:39,771 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,771 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:39,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:39,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:39,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:39,775 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:39,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:39,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:39,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:39,776 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:39,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,779 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,781 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:39,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,783 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,783 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:39,783 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/3c866bea86a8401ebc431c1252862998 2023-04-22 14:54:39,883 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:39,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:39,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:39,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:39,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:39,887 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:39,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:39,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:39,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,888 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,892 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:39,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:39,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:39,894 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,894 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:39,894 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:39,894 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:39,894 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:39,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:39,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:39,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:39,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a309fcf[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:39,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:39,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:39,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:39,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:39,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:39,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:39,909 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,909 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:39,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:39,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:39,912 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:39,912 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:39,912 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:39,912 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:39,912 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:39,914 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,914 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:39,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:39,914 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:39,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,916 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:39,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:39,919 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:39,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:39,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:39,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:39,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:39,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:39,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:39,919 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:39,919 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:40,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175340013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175340014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175340014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175340016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175340017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,019 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:40,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:40,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:40,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:40,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:40,023 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:40,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:40,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:40,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,024 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:40,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:40,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,025 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:40,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:40,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:40,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:40,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:40,028 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:40,028 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:40,028 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:40,028 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:40,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:40,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,030 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,030 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd68b0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:40,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:40,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,035 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:40,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:40,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:40,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:40,036 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,036 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:40,036 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,036 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:40,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:40,039 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:40,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:40,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:40,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:40,039 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:40,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,041 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:40,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:40,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:40,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,045 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:40,046 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:40,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:40,046 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,046 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:40,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:40,146 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:40,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:40,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:40,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:40,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:40,150 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:40,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:40,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:40,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:40,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:40,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:40,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:40,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:40,154 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:40,155 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:40,155 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:40,155 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:40,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:40,159 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:40,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7fccc9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:40,159 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:40,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:40,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:40,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:40,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,161 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,161 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:40,164 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:40,164 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:40,164 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:40,164 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:40,164 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:40,167 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,167 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:40,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,167 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:40,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:40,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:40,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:40,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:40,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:40,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,173 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,173 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:40,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:40,173 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,173 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:40,173 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:40,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/f8a8c6fa60e34153978f3f7fb37b0dbd 2023-04-22 14:54:40,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/9c717f8ccee4497e9d0854bf2a877e00 2023-04-22 14:54:40,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/3c866bea86a8401ebc431c1252862998 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998 2023-04-22 14:54:40,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998, entries=150, sequenceid=15, filesize=11.6 K 2023-04-22 14:54:40,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/f8a8c6fa60e34153978f3f7fb37b0dbd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd 2023-04-22 14:54:40,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd, entries=150, sequenceid=15, filesize=11.6 K 2023-04-22 14:54:40,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/9c717f8ccee4497e9d0854bf2a877e00 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00 2023-04-22 14:54:40,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00, entries=150, sequenceid=15, filesize=11.6 K 2023-04-22 14:54:40,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 5adf52822e4e313e73f3d2c863d10c45 in 914ms, sequenceid=15, compaction requested=false 2023-04-22 14:54:40,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:40,273 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:40,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:40,277 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:40,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:40,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:40,278 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,278 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,279 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:40,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:40,279 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,279 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,279 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:40,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:40,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:40,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:40,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:40,294 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:40,295 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:40,295 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:40,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:40,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:40,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175340524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175340524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175340525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175340525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175340527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175340628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175340628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175340629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175340630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175340631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,714 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/a3ffb7b7f10b42efb73051eaf52f0ce6 2023-04-22 14:54:40,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175340831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175340831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175340831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175340833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:40,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:40,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175340834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,128 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c24176d292c44257befc6e56f0f62f36 2023-04-22 14:54:41,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175341133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175341135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175341136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175341136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175341139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,541 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/c46e94bbcb634985a94568c7a1f99f5b 2023-04-22 14:54:41,545 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/a3ffb7b7f10b42efb73051eaf52f0ce6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6 2023-04-22 14:54:41,551 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6, entries=150, sequenceid=35, filesize=11.6 K 2023-04-22 14:54:41,552 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c24176d292c44257befc6e56f0f62f36 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36 2023-04-22 14:54:41,560 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36, entries=150, sequenceid=35, filesize=11.6 K 2023-04-22 14:54:41,561 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/c46e94bbcb634985a94568c7a1f99f5b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b 2023-04-22 14:54:41,565 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b, entries=150, sequenceid=35, filesize=11.6 K 2023-04-22 14:54:41,566 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5adf52822e4e313e73f3d2c863d10c45 in 1271ms, sequenceid=35, compaction requested=false 2023-04-22 14:54:41,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:41,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:41,570 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:41,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a2151d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:41,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:41,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,575 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:41,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:41,575 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:41,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:41,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:41,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:41,577 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:41,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:41,580 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:41,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:41,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:41,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:41,580 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:41,582 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,582 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,582 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,583 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,585 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,587 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,587 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,588 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:41,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:41,588 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,588 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:41,588 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:41,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:41,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:41,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:41,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:41,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175341678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175341678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175341678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175341679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175341679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,688 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:41,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:41,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:41,691 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:41,691 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,691 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:41,691 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:41,692 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,692 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:41,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:41,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:41,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:41,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:41,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:41,696 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,697 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,697 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:41,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69e220c1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:41,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:41,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,700 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:41,700 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,700 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:41,703 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:41,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:41,705 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:41,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:41,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:41,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:41,705 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:41,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,708 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,709 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,711 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:41,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:41,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,712 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:41,712 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:41,712 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175341783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175341783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175341783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175341783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175341783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,812 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:41,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:41,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:41,816 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:41,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:41,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:41,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,817 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,818 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,818 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,818 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:41,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:41,820 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,820 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:41,820 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:41,820 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:41,820 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:41,823 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:41,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733da5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:41,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:41,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,829 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:41,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:41,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:41,830 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,830 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:41,831 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:41,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:41,834 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:41,834 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:41,834 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:41,834 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:41,834 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:41,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:41,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,843 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,843 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:41,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:41,843 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,943 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:41,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:41,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:41,947 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:41,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:41,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:41,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:41,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:41,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:41,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:41,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:41,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:41,951 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,951 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:41,951 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:41,951 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:41,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:41,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:41,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442ea0e6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:41,955 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:41,959 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,959 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,960 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:41,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:41,963 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:41,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:41,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:41,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:41,963 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:41,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:41,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:41,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:41,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:41,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:41,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:41,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:41,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:41,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:41,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:41,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:41,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:41,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:41,976 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:41,976 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:41,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175341985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175341986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175341986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175341987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:41,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:41,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175341987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/897f1753f2d6467488bc3dc0c85ec9f6 2023-04-22 14:54:42,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/b17a5b69eebd4c2dab17f882854b5310 2023-04-22 14:54:42,076 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:42,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:42,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:42,081 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:42,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:42,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:42,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:42,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:42,085 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:42,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:42,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:42,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:42,100 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,100 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:42,100 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:42,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a81e8a64035744c8adbb7e32c7bbc2e7 2023-04-22 14:54:42,100 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,101 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:42,101 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:42,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:42,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:42,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d35f83[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:42,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:42,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/897f1753f2d6467488bc3dc0c85ec9f6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6 2023-04-22 14:54:42,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:42,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:42,109 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:42,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:42,116 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:42,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:42,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:42,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:42,116 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:42,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 14:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:42,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/b17a5b69eebd4c2dab17f882854b5310 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310 2023-04-22 14:54:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:42,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:42,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:42,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:42,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:42,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:42,126 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:42,126 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:42,126 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:42,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:42,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:42,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 14:54:42,126 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,128 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,128 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:42,128 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a81e8a64035744c8adbb7e32c7bbc2e7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7 2023-04-22 14:54:42,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7, entries=150, sequenceid=48, filesize=11.6 K 2023-04-22 14:54:42,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5adf52822e4e313e73f3d2c863d10c45 in 493ms, sequenceid=48, compaction requested=true 2023-04-22 14:54:42,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:42,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:42,134 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:42,134 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:42,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:42,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:42,135 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:42,135 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:42,136 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,136 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=34.8 K 2023-04-22 14:54:42,136 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:42,136 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:42,136 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,136 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=34.8 K 2023-04-22 14:54:42,136 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3c866bea86a8401ebc431c1252862998, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682175279350 2023-04-22 14:54:42,138 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f8a8c6fa60e34153978f3f7fb37b0dbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682175279350 2023-04-22 14:54:42,138 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a3ffb7b7f10b42efb73051eaf52f0ce6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682175279370 2023-04-22 14:54:42,138 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c24176d292c44257befc6e56f0f62f36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682175279370 2023-04-22 14:54:42,138 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 897f1753f2d6467488bc3dc0c85ec9f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:42,138 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b17a5b69eebd4c2dab17f882854b5310, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:42,146 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#191 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:42,149 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:42,169 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/895ece500dde496481a4738f036e0e81 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/895ece500dde496481a4738f036e0e81 2023-04-22 14:54:42,175 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into 895ece500dde496481a4738f036e0e81(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:42,175 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:42,175 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=13, startTime=1682175282134; duration=0sec 2023-04-22 14:54:42,176 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:42,176 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:42,177 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:42,177 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:42,178 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,178 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=34.8 K 2023-04-22 14:54:42,178 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9c717f8ccee4497e9d0854bf2a877e00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682175279350 2023-04-22 14:54:42,179 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c46e94bbcb634985a94568c7a1f99f5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682175279370 2023-04-22 14:54:42,179 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a81e8a64035744c8adbb7e32c7bbc2e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:42,183 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/9362779feaf2416fa15ac106d2f4d394 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9362779feaf2416fa15ac106d2f4d394 2023-04-22 14:54:42,189 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 9362779feaf2416fa15ac106d2f4d394(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:42,189 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:42,189 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=13, startTime=1682175282134; duration=0sec 2023-04-22 14:54:42,189 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:42,193 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:42,203 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/ecffb1ad887449598cedb66b681abc85 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ecffb1ad887449598cedb66b681abc85 2023-04-22 14:54:42,209 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into ecffb1ad887449598cedb66b681abc85(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:42,209 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:42,209 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=13, startTime=1682175282134; duration=0sec 2023-04-22 14:54:42,209 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:42,226 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:42,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:42,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:42,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:42,230 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:42,230 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:42,230 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:42,230 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:42,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:42,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:42,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:42,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:42,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:42,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:42,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:42,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:42,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:42,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:42,256 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:42,256 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:42,256 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:42,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:42,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:42,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:42,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175342297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175342299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175342299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175342300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175342300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,346 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:54:42,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175342400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175342402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175342402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175342402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175342402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175342603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175342604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175342605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175342606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175342606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,679 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/fdbb871837594d07b22afc78dfde0ec6 2023-04-22 14:54:42,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175342907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175342908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175342909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175342911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:42,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:42,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175342911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,092 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/8da379461a514325b75b5a0d303e9004 2023-04-22 14:54:43,106 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/95c6d5bf04c846209fda40b066690c0a 2023-04-22 14:54:43,113 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/fdbb871837594d07b22afc78dfde0ec6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6 2023-04-22 14:54:43,118 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6, entries=150, sequenceid=70, filesize=11.6 K 2023-04-22 14:54:43,120 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/8da379461a514325b75b5a0d303e9004 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004 2023-04-22 14:54:43,125 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004, entries=150, sequenceid=70, filesize=11.6 K 2023-04-22 14:54:43,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/95c6d5bf04c846209fda40b066690c0a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a 2023-04-22 14:54:43,136 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a, entries=150, sequenceid=70, filesize=11.6 K 2023-04-22 14:54:43,137 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 5adf52822e4e313e73f3d2c863d10c45 in 881ms, sequenceid=70, compaction requested=false 2023-04-22 14:54:43,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:43,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:43,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:43,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4f8510[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:43,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:43,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:43,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:43,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:43,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:43,154 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:43,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:43,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:43,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:43,154 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:43,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:43,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:43,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:43,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:43,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:43,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:43,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:43,161 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:43,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:43,161 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:43,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:43,261 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:43,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:43,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:43,264 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:43,264 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:43,264 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:43,264 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:43,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:43,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:43,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:43,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:43,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:43,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:43,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:43,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:43,282 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:43,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:43,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:43,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:43,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:43,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:43,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:43,305 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/52aa96f9f44a4db098ecc3804230cdf2 2023-04-22 14:54:43,322 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/281b28f0e9024a11b12320812e2f0203 2023-04-22 14:54:43,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:43,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:43,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175343425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175343426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175343427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175343429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175343430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175343530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175343531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175343531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175343533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175343533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175343733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175343733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175343734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175343735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,737 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a4b7b8bd71ad48809d26f07d359e1af5 2023-04-22 14:54:43,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175343736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,741 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/52aa96f9f44a4db098ecc3804230cdf2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2 2023-04-22 14:54:43,746 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2, entries=150, sequenceid=84, filesize=11.6 K 2023-04-22 14:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/281b28f0e9024a11b12320812e2f0203 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203 2023-04-22 14:54:43,753 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203, entries=150, sequenceid=84, filesize=11.6 K 2023-04-22 14:54:43,754 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a4b7b8bd71ad48809d26f07d359e1af5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5 2023-04-22 14:54:43,759 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5, entries=150, sequenceid=84, filesize=11.6 K 2023-04-22 14:54:43,760 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 478ms, sequenceid=84, compaction requested=true 2023-04-22 14:54:43,760 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:43,760 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:43,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:43,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524817c9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:43,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:43,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,776 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:43,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:43,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:43,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:43,785 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:43,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:43,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:43,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:43,785 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:43,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:43,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:43,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:43,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:43,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,791 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:43,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:43,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:43,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:43,791 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:43,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:43,792 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:43,892 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:43,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:43,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:43,897 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:43,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:43,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:43,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:43,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:43,898 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:43,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:43,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:43,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:43,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:43,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:43,900 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:43,900 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:43,900 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:43,900 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:43,900 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:43,901 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:43,901 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:43,901 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:43,901 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:44,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:44,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:44,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175344046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175344048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175344048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175344049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175344049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175344150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175344152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175344152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175344152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175344153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,312 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/1ff5be1adac2493f9dac276180802444 2023-04-22 14:54:44,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175344352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175344354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175344356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175344356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175344356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175344655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175344657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175344658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175344659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175344661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,724 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c292f79ddb8f4e4096f76740334e420e 2023-04-22 14:54:44,735 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/fdc383869bdb443db74b2917ca4e4ead 2023-04-22 14:54:44,739 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/1ff5be1adac2493f9dac276180802444 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444 2023-04-22 14:54:44,744 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 14:54:44,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c292f79ddb8f4e4096f76740334e420e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e 2023-04-22 14:54:44,749 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 14:54:44,749 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/fdc383869bdb443db74b2917ca4e4ead as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead 2023-04-22 14:54:44,753 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead, entries=150, sequenceid=102, filesize=11.6 K 2023-04-22 14:54:44,754 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 5adf52822e4e313e73f3d2c863d10c45 in 854ms, sequenceid=102, compaction requested=true 2023-04-22 14:54:44,754 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:44,754 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:44,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6350a4f6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:44,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:44,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:44,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:44,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:44,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:44,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:44,763 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:44,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:44,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:44,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:44,763 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:44,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:44,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:44,770 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,770 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:44,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,870 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:44,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:44,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:44,874 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:44,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:44,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:44,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:44,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:44,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,877 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:44,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:44,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:44,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:44,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:44,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:44,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:44,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:44,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:44,878 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:44,879 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:44,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:44,887 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/cceb9c728a974c648e252f33f473b1d9 2023-04-22 14:54:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:45,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:45,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175345172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175345172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175345174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175345175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175345175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175345276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175345276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175345277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175345279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175345279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,299 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/777cdaf8c8264efeaf2f8fd3426ec685 2023-04-22 14:54:45,311 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/fa6d311a09ee4841b1b881e59ec3fefd 2023-04-22 14:54:45,315 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/cceb9c728a974c648e252f33f473b1d9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9 2023-04-22 14:54:45,319 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 14:54:45,320 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/777cdaf8c8264efeaf2f8fd3426ec685 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685 2023-04-22 14:54:45,327 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 14:54:45,328 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/fa6d311a09ee4841b1b881e59ec3fefd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd 2023-04-22 14:54:45,333 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd, entries=150, sequenceid=117, filesize=11.6 K 2023-04-22 14:54:45,334 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 455ms, sequenceid=117, compaction requested=true 2023-04-22 14:54:45,334 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:45,334 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:45,335 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,335 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:45,335 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:45,336 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,336 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a56c058[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:45,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:45,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,340 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:45,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:45,344 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:45,347 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:45,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:45,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:45,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:45,347 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:45,348 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,349 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,349 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:45,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:45,349 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:45,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,353 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:45,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:45,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,353 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,353 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:45,353 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:45,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:45,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:45,454 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:45,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:45,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:45,457 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:45,457 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:45,457 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:45,458 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:45,458 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,458 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:45,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:45,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:45,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:45,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:45,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:45,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:45,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:45,462 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:45,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:45,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:45,462 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:45,463 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:45,463 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:45,463 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:45,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:45,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:45,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175345487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175345487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175345488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175345490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175345491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175345592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175345592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175345592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175345595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175345595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175345795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175345795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175345795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175345798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:45,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175345798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,878 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/27975ed489ba4f49a701b8e0c0a5285f 2023-04-22 14:54:45,906 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/1124cb30531e4504aa2992dffdf9cbbd 2023-04-22 14:54:45,930 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/e5cf52230c7b4944af89b9a03529f8f8 2023-04-22 14:54:45,934 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/27975ed489ba4f49a701b8e0c0a5285f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f 2023-04-22 14:54:45,938 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f, entries=150, sequenceid=135, filesize=11.8 K 2023-04-22 14:54:45,938 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/1124cb30531e4504aa2992dffdf9cbbd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd 2023-04-22 14:54:45,942 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd, entries=150, sequenceid=135, filesize=11.8 K 2023-04-22 14:54:45,943 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/e5cf52230c7b4944af89b9a03529f8f8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8 2023-04-22 14:54:45,947 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8, entries=150, sequenceid=135, filesize=11.8 K 2023-04-22 14:54:45,947 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5adf52822e4e313e73f3d2c863d10c45 in 485ms, sequenceid=135, compaction requested=true 2023-04-22 14:54:45,947 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:45,948 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:45,950 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:45,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d86ab07[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:45,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:45,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:45,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:45,955 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:45,958 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:45,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:45,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:45,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:45,958 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:45,965 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,965 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,965 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:45,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:45,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:45,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:45,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:45,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:45,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:45,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:45,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:45,973 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:45,973 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:46,073 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:46,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:46,077 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:46,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:46,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:46,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:46,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:46,080 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,080 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:46,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:46,080 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:46,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:46,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:46,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:46,083 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:46,083 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:46,084 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:46,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:46,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:46,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:46,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175346112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175346116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175346117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175346118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175346118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175346219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175346222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175346223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175346224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175346224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175346423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175346425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175346427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175346428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175346428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,494 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/2ad179cfcc524550b0254d8fec393e57 2023-04-22 14:54:46,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175346726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175346727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175346732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175346732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175346732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,910 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/ac8808c465b94f1abb183a65cfa15bc6 2023-04-22 14:54:46,922 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/cc2a32728a0c49d3a3aef4cef7d04833 2023-04-22 14:54:46,926 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/2ad179cfcc524550b0254d8fec393e57 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57 2023-04-22 14:54:46,930 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57, entries=150, sequenceid=149, filesize=11.8 K 2023-04-22 14:54:46,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/ac8808c465b94f1abb183a65cfa15bc6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6 2023-04-22 14:54:46,935 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6, entries=150, sequenceid=149, filesize=11.8 K 2023-04-22 14:54:46,936 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/cc2a32728a0c49d3a3aef4cef7d04833 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833 2023-04-22 14:54:46,939 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833, entries=150, sequenceid=149, filesize=11.8 K 2023-04-22 14:54:46,940 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 857ms, sequenceid=149, compaction requested=true 2023-04-22 14:54:46,940 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:46,940 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:46,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:46,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f621d21[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:46,948 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:46,948 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:46,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:46,950 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:46,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:46,953 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:46,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:46,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:46,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:46,953 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:46,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:46,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:46,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:46,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:46,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:46,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:46,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:46,961 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:46,961 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:46,961 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:46,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:46,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:46,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,062 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,065 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,065 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,067 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,068 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,068 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,069 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:47,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:47,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:47,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:47,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,077 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/0e4add539bde473994ac09ecc8c5490a 2023-04-22 14:54:47,086 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/75999ba799e741f997cdba6847bffb5d 2023-04-22 14:54:47,097 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/0c5eca968f29497b98caffc6d8b4f194 2023-04-22 14:54:47,101 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/0e4add539bde473994ac09ecc8c5490a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a 2023-04-22 14:54:47,106 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:54:47,107 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/75999ba799e741f997cdba6847bffb5d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d 2023-04-22 14:54:47,110 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:54:47,111 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/0c5eca968f29497b98caffc6d8b4f194 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194 2023-04-22 14:54:47,114 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:54:47,115 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 5adf52822e4e313e73f3d2c863d10c45 in 46ms, sequenceid=167, compaction requested=true 2023-04-22 14:54:47,115 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:47,115 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,117 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f3632f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,122 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,123 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,123 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,125 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,126 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,130 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,132 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,132 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,232 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,236 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,237 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,237 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:47,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:47,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:47,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b3649[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/339fd7a5725842d386faf98c888867f6 2023-04-22 14:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,252 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,252 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,269 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,269 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175347270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175347270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175287268, max timestamp=1682175287273Num uniques 859; 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175287268, max timestamp=1682175287273Num uniques 859; 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175287268, max timestamp=1682175287273Num uniques 859; 2023-04-22 14:54:47,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:47,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175347302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175347303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175347306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,369 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,372 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,374 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,376 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,376 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,376 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,376 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,377 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175347376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175347376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4d6576[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,385 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,385 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,387 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,388 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,390 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,391 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,393 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,394 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,394 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,394 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,394 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,394 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175347411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175347411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175347413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,494 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,497 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,501 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4239fceb[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,510 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,510 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,510 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,510 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,512 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,512 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,518 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,518 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,518 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,519 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,520 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,522 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,522 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,522 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,522 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,523 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,523 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,522 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175347579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175347580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175347615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175347616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175347619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,623 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,626 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50411598[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,647 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,647 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,647 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/930bd97959ca4ed7bcf755b7b0d2aeb0 2023-04-22 14:54:47,747 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,751 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,756 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8488a7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,763 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,765 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,768 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,768 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,769 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,769 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,769 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,771 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,773 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,774 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,774 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,774 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,773 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,774 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,874 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:47,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:47,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:47,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:47,878 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:47,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:47,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:47,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,879 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,880 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,880 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:47,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:47,881 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,881 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:47,881 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:47,881 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,882 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:47,882 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175347883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab43494[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:47,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:47,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:47,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:47,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,889 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:47,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175347885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,889 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,889 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:47,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:47,891 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:47,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:47,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:47,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:47,891 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:47,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:47,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:47,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:47,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:47,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:47,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:47,899 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:47,899 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:47,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:47,899 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:47,899 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:47,899 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:47,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:47,899 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:47,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175347919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175347919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:47,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175347924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:47,999 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:48,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:48,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:48,003 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:48,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:48,006 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:48,006 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,006 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:48,007 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,007 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:48,007 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:48,007 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:48,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27674b8d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:48,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:48,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,011 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:48,011 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:48,011 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:48,011 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,012 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:48,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:48,016 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:48,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:48,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:48,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:48,016 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:48,018 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,018 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,018 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,019 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:48,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:48,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:48,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,023 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,023 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,023 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,023 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,023 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:48,023 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:48,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a5d5e53020b74f4a861bde2fe3465b88 2023-04-22 14:54:48,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/339fd7a5725842d386faf98c888867f6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6 2023-04-22 14:54:48,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6, entries=200, sequenceid=178, filesize=14.1 K 2023-04-22 14:54:48,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/930bd97959ca4ed7bcf755b7b0d2aeb0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0 2023-04-22 14:54:48,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0, entries=150, sequenceid=178, filesize=11.8 K 2023-04-22 14:54:48,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a5d5e53020b74f4a861bde2fe3465b88 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88 2023-04-22 14:54:48,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88, entries=150, sequenceid=178, filesize=11.8 K 2023-04-22 14:54:48,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5adf52822e4e313e73f3d2c863d10c45 in 851ms, sequenceid=178, compaction requested=true 2023-04-22 14:54:48,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:48,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:48,092 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 14:54:48,092 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 14:54:48,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:48,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:48,097 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110067 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 14:54:48,098 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:48,098 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,098 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/895ece500dde496481a4738f036e0e81, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=107.5 K 2023-04-22 14:54:48,098 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 895ece500dde496481a4738f036e0e81, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:48,099 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fdbb871837594d07b22afc78dfde0ec6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175281654 2023-04-22 14:54:48,099 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 52aa96f9f44a4db098ecc3804230cdf2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682175282296 2023-04-22 14:54:48,099 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1ff5be1adac2493f9dac276180802444, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682175283419 2023-04-22 14:54:48,100 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting cceb9c728a974c648e252f33f473b1d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175284047 2023-04-22 14:54:48,100 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 27975ed489ba4f49a701b8e0c0a5285f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175285173 2023-04-22 14:54:48,101 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 2ad179cfcc524550b0254d8fec393e57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682175285486 2023-04-22 14:54:48,101 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0e4add539bde473994ac09ecc8c5490a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175286109 2023-04-22 14:54:48,101 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107677 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 14:54:48,101 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:48,101 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,101 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 339fd7a5725842d386faf98c888867f6, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:48,101 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9362779feaf2416fa15ac106d2f4d394, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=105.2 K 2023-04-22 14:54:48,102 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9362779feaf2416fa15ac106d2f4d394, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:48,104 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8da379461a514325b75b5a0d303e9004, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175281654 2023-04-22 14:54:48,104 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 281b28f0e9024a11b12320812e2f0203, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682175282296 2023-04-22 14:54:48,105 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c292f79ddb8f4e4096f76740334e420e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682175283419 2023-04-22 14:54:48,105 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 777cdaf8c8264efeaf2f8fd3426ec685, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175284047 2023-04-22 14:54:48,106 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1124cb30531e4504aa2992dffdf9cbbd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175285173 2023-04-22 14:54:48,106 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ac8808c465b94f1abb183a65cfa15bc6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682175285486 2023-04-22 14:54:48,106 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 75999ba799e741f997cdba6847bffb5d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175286109 2023-04-22 14:54:48,107 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 930bd97959ca4ed7bcf755b7b0d2aeb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:48,118 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#218 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:48,123 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:48,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:48,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:48,128 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#219 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:48,132 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:48,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:48,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:48,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,134 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:48,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:48,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:48,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:48,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:48,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:48,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:48,136 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:48,137 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:48,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:48,142 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/c57b3e510c6e495f98672e34e9ef6510 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/c57b3e510c6e495f98672e34e9ef6510 2023-04-22 14:54:48,147 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into c57b3e510c6e495f98672e34e9ef6510(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:48,147 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:48,147 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=7, startTime=1682175288092; duration=0sec 2023-04-22 14:54:48,147 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:48,147 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/5cfc594d644645a49f1dc027ad5ea73d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5cfc594d644645a49f1dc027ad5ea73d 2023-04-22 14:54:48,147 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-22 14:54:48,150 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/077be341a0b549c084438eab9c8c8c99 2023-04-22 14:54:48,151 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107677 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-22 14:54:48,151 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:48,151 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,151 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ecffb1ad887449598cedb66b681abc85, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=105.2 K 2023-04-22 14:54:48,151 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ecffb1ad887449598cedb66b681abc85, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682175280523 2023-04-22 14:54:48,152 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 95c6d5bf04c846209fda40b066690c0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175281654 2023-04-22 14:54:48,152 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a4b7b8bd71ad48809d26f07d359e1af5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682175282296 2023-04-22 14:54:48,152 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 5cfc594d644645a49f1dc027ad5ea73d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:48,153 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fdc383869bdb443db74b2917ca4e4ead, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682175283419 2023-04-22 14:54:48,153 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:48,153 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=7, startTime=1682175288092; duration=0sec 2023-04-22 14:54:48,153 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:48,153 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fa6d311a09ee4841b1b881e59ec3fefd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682175284047 2023-04-22 14:54:48,154 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e5cf52230c7b4944af89b9a03529f8f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175285173 2023-04-22 14:54:48,154 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting cc2a32728a0c49d3a3aef4cef7d04833, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682175285486 2023-04-22 14:54:48,155 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0c5eca968f29497b98caffc6d8b4f194, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175286109 2023-04-22 14:54:48,155 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a5d5e53020b74f4a861bde2fe3465b88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:48,164 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/662037dbf62b423eac0e34a0b9ef946a 2023-04-22 14:54:48,183 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#223 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:48,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:48,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:48,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175348407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175348407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175348427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175348427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175348430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175348512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175348513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,586 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/4407633b8be34557a53a43784ae1e1ab 2023-04-22 14:54:48,591 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/077be341a0b549c084438eab9c8c8c99 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99 2023-04-22 14:54:48,594 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8f99e9669fd24ba9a8688233a7d0c8ad as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8f99e9669fd24ba9a8688233a7d0c8ad 2023-04-22 14:54:48,595 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99, entries=150, sequenceid=214, filesize=11.8 K 2023-04-22 14:54:48,602 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/662037dbf62b423eac0e34a0b9ef946a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a 2023-04-22 14:54:48,606 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into 8f99e9669fd24ba9a8688233a7d0c8ad(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:48,606 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:48,606 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=7, startTime=1682175288093; duration=0sec 2023-04-22 14:54:48,606 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:48,608 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a, entries=150, sequenceid=214, filesize=11.8 K 2023-04-22 14:54:48,611 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/4407633b8be34557a53a43784ae1e1ab as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab 2023-04-22 14:54:48,615 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab, entries=150, sequenceid=214, filesize=11.8 K 2023-04-22 14:54:48,616 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 5adf52822e4e313e73f3d2c863d10c45 in 479ms, sequenceid=214, compaction requested=false 2023-04-22 14:54:48,616 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:48,616 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:48,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:48,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd2772c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:48,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:48,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,626 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:48,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:48,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:48,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:48,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:48,631 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:48,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:48,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:48,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:48,631 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:48,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:48,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,640 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:48,640 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:48,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:48,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:48,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:48,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/7c35eae04e774356ba9ffa946c201239 2023-04-22 14:54:48,741 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:48,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:48,744 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:48,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:48,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:48,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,745 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:48,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:48,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:48,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175348748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175348749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:48,753 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,753 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:48,753 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:48,753 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:48,757 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274b09c1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:48,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:48,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:48,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:48,762 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:48,764 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:48,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:48,765 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:48,765 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:48,765 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:48,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:48,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:48,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:48,771 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:48,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,771 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:48,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175348852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:48,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175348852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,871 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:48,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:48,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:48,874 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:48,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:48,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:48,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:48,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:48,878 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,878 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:48,878 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:48,878 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:48,881 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd08928[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:48,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:48,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:48,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:48,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:48,884 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,884 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:48,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:48,888 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:48,888 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:48,888 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:48,888 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:48,888 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:48,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:48,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:48,890 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:48,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:48,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:48,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:48,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:48,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,895 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:48,895 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:48,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:48,996 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:48,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:48,999 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:48,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:48,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:48,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:48,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:48,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,001 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:49,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:49,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:49,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:49,003 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,003 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:49,003 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:49,003 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:49,005 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,005 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,005 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:49,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:49,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f6841c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:49,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:49,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:49,015 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:49,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:49,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:49,015 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:49,015 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:49,017 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,017 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,017 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,019 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:49,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:49,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:49,021 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:49,021 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:49,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51760 deadline: 1682175349055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51730 deadline: 1682175349056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,122 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:49,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:49,125 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:49,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:49,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:49,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:49,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:49,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:49,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:49,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:49,130 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d71ee19[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:49,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:49,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:49,134 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,134 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:49,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:49,137 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:49,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:49,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:49,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:49,138 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:49,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/ae084cb176d34845b2ec9a49d5ee3c6a 2023-04-22 14:54:49,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:49,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:49,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:49,143 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:49,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:49,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:49,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:49,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:49,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:49,150 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:49,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:49,150 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:49,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:49,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:49,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:49,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/50e57b447d52447188c312776383fdc7 2023-04-22 14:54:49,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/7c35eae04e774356ba9ffa946c201239 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239 2023-04-22 14:54:49,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239, entries=150, sequenceid=231, filesize=11.8 K 2023-04-22 14:54:49,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/ae084cb176d34845b2ec9a49d5ee3c6a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a 2023-04-22 14:54:49,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a, entries=150, sequenceid=231, filesize=11.8 K 2023-04-22 14:54:49,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/50e57b447d52447188c312776383fdc7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7 2023-04-22 14:54:49,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7, entries=150, sequenceid=231, filesize=11.8 K 2023-04-22 14:54:49,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 474ms, sequenceid=231, compaction requested=true 2023-04-22 14:54:49,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:49,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:49,192 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:49,192 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:49,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:49,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:49,193 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:49,193 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:49,193 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:49,193 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:49,193 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,193 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,193 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/c57b3e510c6e495f98672e34e9ef6510, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=35.7 K 2023-04-22 14:54:49,193 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5cfc594d644645a49f1dc027ad5ea73d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=35.7 K 2023-04-22 14:54:49,194 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c57b3e510c6e495f98672e34e9ef6510, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:49,194 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5cfc594d644645a49f1dc027ad5ea73d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:49,194 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 077be341a0b549c084438eab9c8c8c99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682175287282 2023-04-22 14:54:49,194 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 662037dbf62b423eac0e34a0b9ef946a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682175287282 2023-04-22 14:54:49,195 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ae084cb176d34845b2ec9a49d5ee3c6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:49,195 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7c35eae04e774356ba9ffa946c201239, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:49,202 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:49,203 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:49,226 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/585e5e6d9f1d46a39cefd14df3205c2b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/585e5e6d9f1d46a39cefd14df3205c2b 2023-04-22 14:54:49,226 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/299f02d7dcc1474cb46b3042a418c360 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/299f02d7dcc1474cb46b3042a418c360 2023-04-22 14:54:49,232 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 299f02d7dcc1474cb46b3042a418c360(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:49,232 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into 585e5e6d9f1d46a39cefd14df3205c2b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:49,232 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:49,232 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:49,232 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=13, startTime=1682175289192; duration=0sec 2023-04-22 14:54:49,232 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=13, startTime=1682175289192; duration=0sec 2023-04-22 14:54:49,232 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:49,232 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:49,232 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:49,233 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:49,233 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:49,233 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,233 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8f99e9669fd24ba9a8688233a7d0c8ad, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=35.7 K 2023-04-22 14:54:49,234 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8f99e9669fd24ba9a8688233a7d0c8ad, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682175287235 2023-04-22 14:54:49,234 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4407633b8be34557a53a43784ae1e1ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682175287282 2023-04-22 14:54:49,236 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 50e57b447d52447188c312776383fdc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:49,242 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:49,261 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:49,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:49,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:49,265 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:49,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:49,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:49,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:49,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,267 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:49,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:49,268 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/53d430d0ee51492a9983211da81ee7fa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/53d430d0ee51492a9983211da81ee7fa 2023-04-22 14:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:49,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:49,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:49,269 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:49,269 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:49,269 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:49,269 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:49,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:49,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:49,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:49,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:49,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:49,275 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into 53d430d0ee51492a9983211da81ee7fa(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:49,275 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:49,275 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=13, startTime=1682175289192; duration=0sec 2023-04-22 14:54:49,275 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:49,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:49,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:49,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175349372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175349372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175349436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175349438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175349439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175349476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175349476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175349679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175349679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,689 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/aac332072a8e4831a8622df4e1d1659d 2023-04-22 14:54:49,704 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/abf37bdae4c443adbf506be7de4bd8ea 2023-04-22 14:54:49,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175349981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:49,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:49,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175349983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,113 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/1a170a64f97b4af096702691912f1445 2023-04-22 14:54:50,118 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/aac332072a8e4831a8622df4e1d1659d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d 2023-04-22 14:54:50,123 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d, entries=150, sequenceid=251, filesize=11.8 K 2023-04-22 14:54:50,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/abf37bdae4c443adbf506be7de4bd8ea as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea 2023-04-22 14:54:50,128 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea, entries=150, sequenceid=251, filesize=11.8 K 2023-04-22 14:54:50,128 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/1a170a64f97b4af096702691912f1445 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445 2023-04-22 14:54:50,132 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445, entries=150, sequenceid=251, filesize=11.8 K 2023-04-22 14:54:50,133 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5adf52822e4e313e73f3d2c863d10c45 in 864ms, sequenceid=251, compaction requested=false 2023-04-22 14:54:50,133 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:50,133 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220be177[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:50,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:50,137 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:50,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,140 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:50,142 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:50,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:50,145 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:50,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:50,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:50,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:50,145 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:50,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:50,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:50,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:50,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:50,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:50,153 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:50,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:50,153 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:50,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:50,253 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:50,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:50,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:50,256 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:50,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:50,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:50,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:50,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:50,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,259 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:50,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:50,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:50,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:50,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:50,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:50,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:50,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:50,261 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:50,261 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:50,261 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:50,261 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:50,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:50,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:50,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:50,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:50,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:50,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:50,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:50,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175350509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175350511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175350612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175350615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,669 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/70376881438746edbc1e557f2c496847 2023-04-22 14:54:50,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175350816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:50,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175350818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,083 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/fe96b5dcef844679b76a5cbdf659130d 2023-04-22 14:54:51,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175351121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175351121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175351441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,446 DEBUG [Thread-1466] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:51,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175351447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,450 DEBUG [Thread-1468] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:51,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175351453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,456 DEBUG [Thread-1464] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:51,494 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/edae34d79a874c4ba8749aae8a6c084f 2023-04-22 14:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/70376881438746edbc1e557f2c496847 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847 2023-04-22 14:54:51,503 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847, entries=150, sequenceid=266, filesize=11.9 K 2023-04-22 14:54:51,504 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/fe96b5dcef844679b76a5cbdf659130d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d 2023-04-22 14:54:51,509 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d, entries=150, sequenceid=266, filesize=11.9 K 2023-04-22 14:54:51,510 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/edae34d79a874c4ba8749aae8a6c084f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f 2023-04-22 14:54:51,514 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f, entries=150, sequenceid=266, filesize=11.9 K 2023-04-22 14:54:51,515 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 1254ms, sequenceid=266, compaction requested=true 2023-04-22 14:54:51,515 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:51,515 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,515 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:51,515 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:51,515 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:51,521 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abab5d7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:51,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:51,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,525 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:51,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:51,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:51,530 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:51,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:51,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:51,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:51,530 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:51,532 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,532 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,533 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,534 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,535 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:51,545 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:51,546 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,546 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,546 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,546 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:51,546 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:51,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:51,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,646 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:51,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/8d0a72d4606a498197038f9f2a7874d3 2023-04-22 14:54:51,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175351646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:51,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:51,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:51,650 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:51,650 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,650 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:51,650 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:51,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175351648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,652 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,652 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,653 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,653 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:51,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:51,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:51,662 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,662 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:51,662 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:51,662 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:51,664 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6906f8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:51,668 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:51,668 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,669 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:51,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,670 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:51,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/a659a7a242004536afc92156b341aa17 2023-04-22 14:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:51,671 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:51,676 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:51,676 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:51,676 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:51,676 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:51,676 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:51,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,681 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,683 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,684 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:51,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,684 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:51,684 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:51,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/89cc6f92b82b4cfa92bdb899ca060ee2 2023-04-22 14:54:51,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/8d0a72d4606a498197038f9f2a7874d3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3 2023-04-22 14:54:51,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3, entries=200, sequenceid=285, filesize=14.3 K 2023-04-22 14:54:51,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/a659a7a242004536afc92156b341aa17 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17 2023-04-22 14:54:51,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17, entries=150, sequenceid=285, filesize=11.9 K 2023-04-22 14:54:51,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/89cc6f92b82b4cfa92bdb899ca060ee2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2 2023-04-22 14:54:51,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2, entries=150, sequenceid=285, filesize=11.9 K 2023-04-22 14:54:51,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5adf52822e4e313e73f3d2c863d10c45 in 75ms, sequenceid=285, compaction requested=true 2023-04-22 14:54:51,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:51,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:51,702 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:51,702 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:51,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:51,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:51,703 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51396 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:51,703 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:51,703 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:51,703 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:51,703 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,703 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,703 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/585e5e6d9f1d46a39cefd14df3205c2b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=50.2 K 2023-04-22 14:54:51,704 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/299f02d7dcc1474cb46b3042a418c360, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=47.8 K 2023-04-22 14:54:51,704 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 585e5e6d9f1d46a39cefd14df3205c2b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:51,704 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 299f02d7dcc1474cb46b3042a418c360, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:51,704 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting aac332072a8e4831a8622df4e1d1659d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175288736 2023-04-22 14:54:51,704 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting abf37bdae4c443adbf506be7de4bd8ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175288736 2023-04-22 14:54:51,708 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 70376881438746edbc1e557f2c496847, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682175289369 2023-04-22 14:54:51,708 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting fe96b5dcef844679b76a5cbdf659130d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682175289369 2023-04-22 14:54:51,708 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8d0a72d4606a498197038f9f2a7874d3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290503 2023-04-22 14:54:51,708 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a659a7a242004536afc92156b341aa17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290508 2023-04-22 14:54:51,717 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:51,720 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:51,742 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/b14a2bf424084ec3bb39c93efd851ee5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/b14a2bf424084ec3bb39c93efd851ee5 2023-04-22 14:54:51,746 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/844a35afa71e4a3fb4925ba7dc9f6b66 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/844a35afa71e4a3fb4925ba7dc9f6b66 2023-04-22 14:54:51,749 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into b14a2bf424084ec3bb39c93efd851ee5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:51,749 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:51,749 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=12, startTime=1682175291702; duration=0sec 2023-04-22 14:54:51,749 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:51,749 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:54:51,750 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:54:51,751 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:51,751 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,751 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/53d430d0ee51492a9983211da81ee7fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=47.8 K 2023-04-22 14:54:51,751 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 53d430d0ee51492a9983211da81ee7fa, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175288396 2023-04-22 14:54:51,752 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1a170a64f97b4af096702691912f1445, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682175288736 2023-04-22 14:54:51,752 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting edae34d79a874c4ba8749aae8a6c084f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682175289369 2023-04-22 14:54:51,752 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 89cc6f92b82b4cfa92bdb899ca060ee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290508 2023-04-22 14:54:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:51,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:51,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:51,767 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 844a35afa71e4a3fb4925ba7dc9f6b66(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:51,767 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:51,767 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#241 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:51,767 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=12, startTime=1682175291702; duration=0sec 2023-04-22 14:54:51,767 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:51,784 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:51,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:51,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175351784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175351785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:51,796 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:51,796 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,796 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:51,796 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:51,796 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,799 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:51,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:51,824 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:51,824 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:51,824 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:51,824 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dba202a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:51,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:51,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,829 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:51,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:51,830 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:51,833 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:51,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:51,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:51,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:51,833 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:51,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,836 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,839 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:51,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,839 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:51,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175351888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175351888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,940 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:51,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:51,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:51,943 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:51,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:51,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:51,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:51,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:51,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:51,947 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,947 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:51,947 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:51,947 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:51,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:51,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c762b3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:51,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:51,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:51,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:51,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:51,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:51,954 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:51,957 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:51,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:51,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:51,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:51,957 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:51,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:51,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:51,959 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:51,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:51,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:51,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:51,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:51,967 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:51,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:51,968 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:51,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:51,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,068 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,071 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,073 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,074 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,075 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,075 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,075 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,075 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:52,075 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:52,076 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:52,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0c6f71[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:52,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:52,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,081 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:52,081 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:52,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:52,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:52,085 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:52,085 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:52,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:52,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:52,086 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:52,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,088 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,089 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175352091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175352091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:52,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,093 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,093 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:52,093 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:52,193 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,197 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,200 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,200 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,200 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,200 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,201 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,202 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,202 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,202 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,202 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,204 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3815f8ae[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:52,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:52,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,207 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:52,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:52,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:52,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:52,208 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/4a1baa18600f4c65b9db63563d227a51 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4a1baa18600f4c65b9db63563d227a51 2023-04-22 14:54:52,210 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/90d7db89036f48eab88454f6229726bb 2023-04-22 14:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,214 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into 4a1baa18600f4c65b9db63563d227a51(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:52,214 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:52,214 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=12, startTime=1682175291702; duration=0sec 2023-04-22 14:54:52,214 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:52,214 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:52,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:52,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:52,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:52,215 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:52,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,218 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,224 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:52,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:52,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,224 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,224 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:52,224 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:52,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,225 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/5aaf7b301b784eb9a921c2f8d200d0e3 2023-04-22 14:54:52,324 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,327 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,329 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,330 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,330 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:52,333 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a234f6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:52,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:52,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,336 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:52,337 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:52,337 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,337 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:52,338 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:52,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:52,340 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:52,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:52,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:52,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:52,340 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,344 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,346 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:52,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:52,346 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,346 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:52,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175352394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175352395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,446 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,450 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,455 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,456 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,456 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,457 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,458 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,458 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,458 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,458 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:52,461 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,461 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feb1a2b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:52,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:52,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,466 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:52,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,466 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,467 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:52,470 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:52,470 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:52,470 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:52,470 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:52,470 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:52,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,473 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,474 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:52,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:52,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:52,477 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:52,477 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:52,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,577 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,588 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,588 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,588 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,588 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,591 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:52,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,596 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,596 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,596 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,596 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:52,596 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:52,598 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,598 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:52,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33809f40[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:52,599 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:52,599 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,600 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,601 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:52,601 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:52,601 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:52,601 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,601 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:52,602 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,602 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:52,602 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,602 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:52,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:52,604 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:52,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:52,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:52,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:52,604 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:52,607 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,607 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:52,607 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,609 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,612 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:52,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:52,613 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:52,613 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:52,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8d2f676782254f60888f9c48011946cd 2023-04-22 14:54:52,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/90d7db89036f48eab88454f6229726bb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb 2023-04-22 14:54:52,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb, entries=150, sequenceid=300, filesize=11.9 K 2023-04-22 14:54:52,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/5aaf7b301b784eb9a921c2f8d200d0e3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3 2023-04-22 14:54:52,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3, entries=150, sequenceid=300, filesize=11.9 K 2023-04-22 14:54:52,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8d2f676782254f60888f9c48011946cd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd 2023-04-22 14:54:52,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd, entries=150, sequenceid=300, filesize=11.9 K 2023-04-22 14:54:52,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 904ms, sequenceid=300, compaction requested=false 2023-04-22 14:54:52,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:52,713 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:52,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:52,716 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:52,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:52,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:52,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,718 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:52,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:52,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:52,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:52,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:52,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:52,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:52,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:52,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:52,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:52,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:52,720 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:52,720 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:52,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:52,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:52,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175352911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:52,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175352911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175353013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175353014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,127 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/08c4db1256d24aceb69f149e41680abe 2023-04-22 14:54:53,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175353216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175353217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175353519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:53,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175353521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,538 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/f5fad8a7e1e44f05b4bed2fbac9a08f9 2023-04-22 14:54:53,548 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/03d2b22b059a449c972d0c25ba35c647 2023-04-22 14:54:53,552 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/08c4db1256d24aceb69f149e41680abe as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe 2023-04-22 14:54:53,555 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe, entries=150, sequenceid=320, filesize=11.9 K 2023-04-22 14:54:53,556 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/f5fad8a7e1e44f05b4bed2fbac9a08f9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9 2023-04-22 14:54:53,559 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9, entries=150, sequenceid=320, filesize=11.9 K 2023-04-22 14:54:53,560 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/03d2b22b059a449c972d0c25ba35c647 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647 2023-04-22 14:54:53,563 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647, entries=150, sequenceid=320, filesize=11.9 K 2023-04-22 14:54:53,564 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5adf52822e4e313e73f3d2c863d10c45 in 844ms, sequenceid=320, compaction requested=true 2023-04-22 14:54:53,564 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:53,564 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,566 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c7d116[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:53,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:53,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,570 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:53,570 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,570 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:53,572 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:53,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:53,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:53,574 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:53,574 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:53,575 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:53,575 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:53,575 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:53,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:53,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:53,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,578 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:53,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:53,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:53,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:53,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:53,581 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:53,581 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,581 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:53,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:53,581 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:53,681 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:53,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:53,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:53,684 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:53,684 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:53,684 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:53,684 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:53,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:53,687 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:53,687 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:53,688 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:53,688 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:53,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:53,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:53,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:53,689 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:53,689 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:53,689 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:53,690 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:53,697 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/63cd01aed7f3405891c4b0dd39813559 2023-04-22 14:54:53,707 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/05eb5816814b464c9377b6fec6b829b5 2023-04-22 14:54:54,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:54,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175354044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175354046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,119 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/d9102e7f1b0e46ec80dfb8337c351f39 2023-04-22 14:54:54,127 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/63cd01aed7f3405891c4b0dd39813559 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559 2023-04-22 14:54:54,131 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559, entries=150, sequenceid=334, filesize=11.9 K 2023-04-22 14:54:54,132 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/05eb5816814b464c9377b6fec6b829b5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5 2023-04-22 14:54:54,136 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5, entries=150, sequenceid=334, filesize=11.9 K 2023-04-22 14:54:54,137 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/d9102e7f1b0e46ec80dfb8337c351f39 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39 2023-04-22 14:54:54,140 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39, entries=150, sequenceid=334, filesize=11.9 K 2023-04-22 14:54:54,141 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 452ms, sequenceid=334, compaction requested=true 2023-04-22 14:54:54,141 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:54,141 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,143 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e754fc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,146 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,148 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:54,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,151 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,151 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,153 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,155 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,155 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,155 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:54,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:54,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,157 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,158 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/1efe24d6329f4430b7608318f52240b8 2023-04-22 14:54:54,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175354171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175354172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,258 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,261 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,261 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,265 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762a2ca9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,270 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,270 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,270 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,270 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,271 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,271 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,272 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,274 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,274 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,275 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,275 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175354275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,276 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175354276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,278 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,281 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:54,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,282 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,282 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,282 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,382 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,386 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,389 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e7d7b3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,394 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,394 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,395 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,397 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,400 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,400 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,400 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,400 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,400 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,401 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,401 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,401 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,402 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,403 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,405 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,405 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:54,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,406 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,406 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,406 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175354477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175354480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,506 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,509 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb2f1f8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,518 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,518 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,519 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,519 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,526 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,526 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,538 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,540 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,540 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,540 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,540 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/35ad69c614784a869ca0545e7fa070c0 2023-04-22 14:54:54,641 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,645 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,646 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,646 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,650 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c90ec6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,653 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,654 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,654 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,657 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,657 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,658 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,658 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,659 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,660 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:54,662 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,662 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,663 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,663 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,763 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,767 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,770 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,774 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47212912[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,777 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,779 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175354780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,782 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,783 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175354783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,791 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,791 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,791 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,791 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,792 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,891 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:54,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:54,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:54,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:54,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:54,895 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:54,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:54,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:54,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,896 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,897 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:54,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:54,898 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,898 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:54,898 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:54,898 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:54,899 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:54,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3804240a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:54,902 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:54,902 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,904 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:54,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:54,905 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:54,908 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:54,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:54,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:54,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:54,908 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:54,909 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,909 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,909 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:54,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,911 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:54,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:54,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:54,917 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:54,916 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:54,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:54,917 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:54,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:54,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:54,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:54,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:54,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:54,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/d5a75dcbcc3c49dc82c54979261705a2 2023-04-22 14:54:54,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/1efe24d6329f4430b7608318f52240b8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8 2023-04-22 14:54:54,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8, entries=150, sequenceid=354, filesize=11.9 K 2023-04-22 14:54:54,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/35ad69c614784a869ca0545e7fa070c0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0 2023-04-22 14:54:54,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0, entries=150, sequenceid=354, filesize=11.9 K 2023-04-22 14:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/d5a75dcbcc3c49dc82c54979261705a2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2 2023-04-22 14:54:55,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2, entries=150, sequenceid=354, filesize=11.9 K 2023-04-22 14:54:55,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5adf52822e4e313e73f3d2c863d10c45 in 853ms, sequenceid=354, compaction requested=true 2023-04-22 14:54:55,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:55,003 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:55,003 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:55,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:55,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:55,005 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:55,005 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:55,005 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:55,005 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,005 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:55,005 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/b14a2bf424084ec3bb39c93efd851ee5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=60.1 K 2023-04-22 14:54:55,005 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,005 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/844a35afa71e4a3fb4925ba7dc9f6b66, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=60.1 K 2023-04-22 14:54:55,006 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting b14a2bf424084ec3bb39c93efd851ee5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290508 2023-04-22 14:54:55,006 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 844a35afa71e4a3fb4925ba7dc9f6b66, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290508 2023-04-22 14:54:55,006 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 90d7db89036f48eab88454f6229726bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682175291644 2023-04-22 14:54:55,006 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5aaf7b301b784eb9a921c2f8d200d0e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682175291644 2023-04-22 14:54:55,007 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f5fad8a7e1e44f05b4bed2fbac9a08f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682175291774 2023-04-22 14:54:55,007 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 08c4db1256d24aceb69f149e41680abe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682175291774 2023-04-22 14:54:55,008 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 05eb5816814b464c9377b6fec6b829b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682175292905 2023-04-22 14:54:55,008 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 63cd01aed7f3405891c4b0dd39813559, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682175292905 2023-04-22 14:54:55,008 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 35ad69c614784a869ca0545e7fa070c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:55,008 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1efe24d6329f4430b7608318f52240b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:55,017 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,020 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#254 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:55,021 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,035 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,045 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,051 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/513a8d9fc0be4b4fbb4bdf0893ccc557 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/513a8d9fc0be4b4fbb4bdf0893ccc557 2023-04-22 14:54:55,055 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/ee1e2f9384264382a156e609252b0395 2023-04-22 14:54:55,056 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 513a8d9fc0be4b4fbb4bdf0893ccc557(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:55,056 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,056 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=11, startTime=1682175295003; duration=0sec 2023-04-22 14:54:55,057 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:55,057 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:54:55,058 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:54:55,058 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:55,059 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,059 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4a1baa18600f4c65b9db63563d227a51, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=60.1 K 2023-04-22 14:54:55,059 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4a1baa18600f4c65b9db63563d227a51, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175290508 2023-04-22 14:54:55,059 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8d2f676782254f60888f9c48011946cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682175291644 2023-04-22 14:54:55,060 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 03d2b22b059a449c972d0c25ba35c647, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1682175291774 2023-04-22 14:54:55,060 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d9102e7f1b0e46ec80dfb8337c351f39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682175292905 2023-04-22 14:54:55,060 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d5a75dcbcc3c49dc82c54979261705a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:55,070 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:55,072 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/e4c0178cf34f4e368ef89414b89cb66b 2023-04-22 14:54:55,080 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a1466d8e658249cb9e44db5f97b0d0c2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a1466d8e658249cb9e44db5f97b0d0c2 2023-04-22 14:54:55,091 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into a1466d8e658249cb9e44db5f97b0d0c2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:55,091 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,091 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=11, startTime=1682175295003; duration=0sec 2023-04-22 14:54:55,091 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:55,092 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/ec400c71d1474c86ae21baac4fec55fd 2023-04-22 14:54:55,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/ee1e2f9384264382a156e609252b0395 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395 2023-04-22 14:54:55,100 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395, entries=150, sequenceid=366, filesize=11.9 K 2023-04-22 14:54:55,101 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/e4c0178cf34f4e368ef89414b89cb66b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b 2023-04-22 14:54:55,105 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b, entries=150, sequenceid=366, filesize=11.9 K 2023-04-22 14:54:55,105 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/ec400c71d1474c86ae21baac4fec55fd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd 2023-04-22 14:54:55,109 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd, entries=150, sequenceid=366, filesize=11.9 K 2023-04-22 14:54:55,110 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 5adf52822e4e313e73f3d2c863d10c45 in 66ms, sequenceid=366, compaction requested=false 2023-04-22 14:54:55,110 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,110 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,112 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f53fc3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,116 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,118 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,120 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,121 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,126 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,126 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,127 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,127 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,227 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,231 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,234 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,234 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,235 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,235 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c62b0a5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,242 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,246 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,246 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,255 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,255 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:55,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175295301, max timestamp=1682175295322Num uniques 859; 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175295301, max timestamp=1682175295322Num uniques 859; 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175295301, max timestamp=1682175295322Num uniques 859; 2023-04-22 14:54:55,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:54:55,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175355352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,355 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175355353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,358 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,360 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,364 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,364 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,364 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,364 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,366 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c56b82[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,369 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,374 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,374 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,375 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,376 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,376 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,376 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,377 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,383 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,383 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,451 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/71df6675490248318e3bb213f914141b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/71df6675490248318e3bb213f914141b 2023-04-22 14:54:55,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51726 deadline: 1682175355450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,453 DEBUG [Thread-1466] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:55,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175355456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,457 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into 71df6675490248318e3bb213f914141b(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:55,457 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:55,457 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=11, startTime=1682175295003; duration=0sec 2023-04-22 14:54:55,457 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175355458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51750 deadline: 1682175355460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,464 DEBUG [Thread-1468] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:55,483 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51682 deadline: 1682175355484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,487 DEBUG [Thread-1464] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:54:55,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,488 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,497 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,497 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,497 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,497 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7829a4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,503 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,505 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,505 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,508 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,511 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,512 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,512 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,612 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,616 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,617 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,619 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,619 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,619 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,619 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ae042f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,626 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,630 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,630 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,636 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,636 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175355658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175355662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/395eb36ba05f4567bdeb5e7f8763ecdf 2023-04-22 14:54:55,737 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,747 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0061c7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,754 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,754 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,757 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,757 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,757 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,761 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,761 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,861 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,865 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,874 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262090de[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:55,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:55,878 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:55,881 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:55,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:55,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:55,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:55,881 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:55,882 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,882 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,883 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,884 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:55,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:55,888 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,888 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:55,888 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:55,888 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:55,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:55,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175355960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:55,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175355968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,988 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:55,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:55,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:55,992 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:55,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:55,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:55,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:55,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:55,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:55,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:55,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:55,997 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdfa8b7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:55,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:55,999 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:56,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:56,004 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:56,006 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:56,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:56,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:56,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:56,006 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:56,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:56,009 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,013 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:56,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,013 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:56,013 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:56,013 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:56,114 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:56,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:56,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/610bb7372b3d4891a78c989c1f618ef7 2023-04-22 14:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:56,117 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:56,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:56,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:56,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:56,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:56,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:56,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:56,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:56,121 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,121 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:56,121 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:56,121 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:56,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:56,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/04c3f52d0852400b8d3824452f48893e 2023-04-22 14:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af6467f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:56,126 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:56,126 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:56,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:56,129 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:56,132 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:56,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:56,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:56,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:56,132 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:56,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:56,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/395eb36ba05f4567bdeb5e7f8763ecdf as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf 2023-04-22 14:54:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:56,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,136 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf, entries=150, sequenceid=379, filesize=11.9 K 2023-04-22 14:54:56,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:56,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/610bb7372b3d4891a78c989c1f618ef7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:56,141 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:56,141 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:56,141 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:56,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:56,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:56,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7, entries=100, sequenceid=379, filesize=9.5 K 2023-04-22 14:54:56,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/04c3f52d0852400b8d3824452f48893e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e 2023-04-22 14:54:56,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e, entries=100, sequenceid=379, filesize=9.5 K 2023-04-22 14:54:56,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5adf52822e4e313e73f3d2c863d10c45 in 857ms, sequenceid=379, compaction requested=true 2023-04-22 14:54:56,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:56,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:56,151 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:56,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:56,152 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:56,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:56,153 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37372 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:56,153 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34932 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:56,153 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:56,153 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:56,153 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,153 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,153 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/71df6675490248318e3bb213f914141b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=36.5 K 2023-04-22 14:54:56,153 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/513a8d9fc0be4b4fbb4bdf0893ccc557, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=34.1 K 2023-04-22 14:54:56,153 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 71df6675490248318e3bb213f914141b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:56,154 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 513a8d9fc0be4b4fbb4bdf0893ccc557, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:56,154 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ee1e2f9384264382a156e609252b0395, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682175294162 2023-04-22 14:54:56,154 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e4c0178cf34f4e368ef89414b89cb66b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682175294162 2023-04-22 14:54:56,154 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 395eb36ba05f4567bdeb5e7f8763ecdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175295291 2023-04-22 14:54:56,154 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 610bb7372b3d4891a78c989c1f618ef7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175295293 2023-04-22 14:54:56,160 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#263 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:56,170 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:56,241 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:56,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:56,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:56,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:56,252 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:56,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:56,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:56,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:56,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,252 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:56,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:56,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:56,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:56,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:56,257 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:56,257 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:56,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:56,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:56,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175356498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175356499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,586 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/5ca61b45e6dd412da65413ab238fcec7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5ca61b45e6dd412da65413ab238fcec7 2023-04-22 14:54:56,587 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/fee573e81e75450c95ce5e3ee7c50380 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fee573e81e75450c95ce5e3ee7c50380 2023-04-22 14:54:56,591 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 5ca61b45e6dd412da65413ab238fcec7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:56,591 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:56,591 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=13, startTime=1682175296152; duration=0sec 2023-04-22 14:54:56,591 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into fee573e81e75450c95ce5e3ee7c50380(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:56,591 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:56,591 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:56,591 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:56,591 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=13, startTime=1682175296151; duration=0sec 2023-04-22 14:54:56,592 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:56,592 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34932 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:56,593 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:56,593 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:56,593 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a1466d8e658249cb9e44db5f97b0d0c2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=34.1 K 2023-04-22 14:54:56,593 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a1466d8e658249cb9e44db5f97b0d0c2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1682175294042 2023-04-22 14:54:56,593 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ec400c71d1474c86ae21baac4fec55fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682175294162 2023-04-22 14:54:56,594 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 04c3f52d0852400b8d3824452f48893e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175295293 2023-04-22 14:54:56,598 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:56,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175356602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175356602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,607 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8ebb19f0aa164d1c97704508230618de as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8ebb19f0aa164d1c97704508230618de 2023-04-22 14:54:56,612 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into 8ebb19f0aa164d1c97704508230618de(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:56,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:56,612 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=13, startTime=1682175296152; duration=0sec 2023-04-22 14:54:56,612 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:56,665 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/564010b25c2e4d928d03200a6cfd86f1 2023-04-22 14:54:56,675 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/65e0e52d785f415eb2cd3c6cf451b101 2023-04-22 14:54:56,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:56,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175356804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:56,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175356805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,086 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/25345795a92d42009394c564ef46dc37 2023-04-22 14:54:57,090 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/564010b25c2e4d928d03200a6cfd86f1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1 2023-04-22 14:54:57,094 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1, entries=150, sequenceid=416, filesize=11.9 K 2023-04-22 14:54:57,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/65e0e52d785f415eb2cd3c6cf451b101 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101 2023-04-22 14:54:57,099 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101, entries=150, sequenceid=416, filesize=11.9 K 2023-04-22 14:54:57,099 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/25345795a92d42009394c564ef46dc37 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37 2023-04-22 14:54:57,103 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37, entries=150, sequenceid=416, filesize=11.9 K 2023-04-22 14:54:57,103 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 5adf52822e4e313e73f3d2c863d10c45 in 846ms, sequenceid=416, compaction requested=false 2023-04-22 14:54:57,103 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:57,104 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:57,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 14:54:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:57,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566418bf[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,112 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,114 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/d3eb0d7426514735a262d5e58297ca39 2023-04-22 14:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,117 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,117 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,117 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:57,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,125 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,125 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c27eb75de84846d3bde015ee11a314e0 2023-04-22 14:54:57,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a30c7f41c1034d0d9682c5efdfbb81ae 2023-04-22 14:54:57,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175357137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175297122, max timestamp=1682175297138Num uniques 859; 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175297122, max timestamp=1682175297138Num uniques 859; 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175297122, max timestamp=1682175297138Num uniques 859; 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:57,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:54:57,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/d3eb0d7426514735a262d5e58297ca39 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39 2023-04-22 14:54:57,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39, entries=200, sequenceid=430, filesize=14.3 K 2023-04-22 14:54:57,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/c27eb75de84846d3bde015ee11a314e0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0 2023-04-22 14:54:57,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0, entries=150, sequenceid=430, filesize=11.9 K 2023-04-22 14:54:57,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/a30c7f41c1034d0d9682c5efdfbb81ae as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae 2023-04-22 14:54:57,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae, entries=150, sequenceid=430, filesize=11.9 K 2023-04-22 14:54:57,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=67.09 KB/68700 for 5adf52822e4e313e73f3d2c863d10c45 in 48ms, sequenceid=430, compaction requested=true 2023-04-22 14:54:57,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:57,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:57,157 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:57,157 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:57,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:57,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:57,158 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:57,158 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:54:57,159 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,159 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fee573e81e75450c95ce5e3ee7c50380, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=39.0 K 2023-04-22 14:54:57,159 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:57,159 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:54:57,159 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,159 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5ca61b45e6dd412da65413ab238fcec7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=36.6 K 2023-04-22 14:54:57,159 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fee573e81e75450c95ce5e3ee7c50380, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175294163 2023-04-22 14:54:57,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:57,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=73.80 KB heapSize=205.69 KB 2023-04-22 14:54:57,160 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5ca61b45e6dd412da65413ab238fcec7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175294163 2023-04-22 14:54:57,160 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 564010b25c2e4d928d03200a6cfd86f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682175295347 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:57,160 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 65e0e52d785f415eb2cd3c6cf451b101, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682175295347 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:57,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:54:57,161 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c27eb75de84846d3bde015ee11a314e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:54:57,161 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d3eb0d7426514735a262d5e58297ca39, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:54:57,170 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:57,170 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:57,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175357218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,225 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,228 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,233 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,233 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,233 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,233 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,234 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e0c4cd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,237 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,237 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,241 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175357241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,242 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,249 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,249 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175357321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,349 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,352 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,353 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,353 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,354 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,355 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,357 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,357 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,357 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,358 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656b50f6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,361 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,362 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,362 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,363 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,365 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,365 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,371 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,371 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175357444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,472 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,475 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,476 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,478 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,478 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,478 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,478 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,480 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,480 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,480 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,480 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,480 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,481 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d56379a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,485 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,485 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,485 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,489 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,491 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,492 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,492 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,492 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,492 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,499 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,499 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175357523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/dbf1a3668eb74306859de7f99cc3a1ad 2023-04-22 14:54:57,586 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/36ac3ab86273401e9691aeb672df743f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/36ac3ab86273401e9691aeb672df743f 2023-04-22 14:54:57,587 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/4165a58171254746a4660536c4920e20 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/4165a58171254746a4660536c4920e20 2023-04-22 14:54:57,592 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into 36ac3ab86273401e9691aeb672df743f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:57,592 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:57,592 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=13, startTime=1682175297157; duration=0sec 2023-04-22 14:54:57,592 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 4165a58171254746a4660536c4920e20(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:57,592 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:57,592 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:57,592 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:54:57,592 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=13, startTime=1682175297157; duration=0sec 2023-04-22 14:54:57,592 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:57,593 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:54:57,593 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:54:57,593 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,593 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8ebb19f0aa164d1c97704508230618de, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=36.6 K 2023-04-22 14:54:57,594 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8ebb19f0aa164d1c97704508230618de, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682175294163 2023-04-22 14:54:57,594 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 25345795a92d42009394c564ef46dc37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682175295347 2023-04-22 14:54:57,594 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a30c7f41c1034d0d9682c5efdfbb81ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:54:57,599 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,601 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:57,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,603 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,605 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,608 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b01756f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,615 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,616 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,618 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,619 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,637 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,637 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:57,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,737 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,741 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,746 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1935026e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,749 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175357747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,753 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,754 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,760 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,760 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,760 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:57,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175357827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,860 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,864 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,868 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,868 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,870 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,870 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26660d5e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,873 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:57,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:57,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:57,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:57,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:57,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,875 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:57,877 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:57,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:57,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:57,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:57,878 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:57,882 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,882 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,883 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,884 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,886 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,886 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,886 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:57,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:57,886 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:57,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:57,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,887 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:57,887 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:57,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:57,887 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/27a82009e1024903b38cf50fcb3b8f64 2023-04-22 14:54:57,987 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:57,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:57,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:57,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:57,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:57,991 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:57,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:57,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:57,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:57,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:57,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:57,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:57,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:57,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:57,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:57,995 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:57,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:57,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2014ef[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:57,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,001 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,004 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,004 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,004 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,006 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,006 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,006 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,011 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,011 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,012 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,012 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,014 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/af3e7f1bdddc44edb784932ccba2ce82 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/af3e7f1bdddc44edb784932ccba2ce82 2023-04-22 14:54:58,020 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into af3e7f1bdddc44edb784932ccba2ce82(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:58,020 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,020 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=13, startTime=1682175297157; duration=0sec 2023-04-22 14:54:58,020 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:58,112 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,115 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,118 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6ab2c9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,128 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,128 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,128 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,134 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,134 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,135 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,135 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,135 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,235 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,239 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,241 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,241 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,243 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,243 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,243 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:58,243 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfdfb98[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,251 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,253 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,253 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51730 deadline: 1682175358252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,255 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,256 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,263 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,263 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:54:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51760 deadline: 1682175358330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,363 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,366 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,368 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:58,370 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,370 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d670144[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,374 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,379 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,381 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,381 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,384 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,387 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,387 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/b2b8f43b78d64376a7ef1e07d15c0c63 2023-04-22 14:54:58,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/dbf1a3668eb74306859de7f99cc3a1ad as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad 2023-04-22 14:54:58,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad, entries=150, sequenceid=458, filesize=11.9 K 2023-04-22 14:54:58,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/27a82009e1024903b38cf50fcb3b8f64 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64 2023-04-22 14:54:58,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64, entries=150, sequenceid=458, filesize=11.9 K 2023-04-22 14:54:58,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/b2b8f43b78d64376a7ef1e07d15c0c63 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63 2023-04-22 14:54:58,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63, entries=150, sequenceid=458, filesize=11.9 K 2023-04-22 14:54:58,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.64 KB/210576, currentSize=107.34 KB/109920 for 5adf52822e4e313e73f3d2c863d10c45 in 1248ms, sequenceid=458, compaction requested=false 2023-04-22 14:54:58,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,487 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,491 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,494 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,495 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:58,495 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:58,503 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/3665561e076f4db790789078bf98e686 2023-04-22 14:54:58,513 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/90e9fbc2771c44a08d476fc74a73fefc 2023-04-22 14:54:58,524 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06 2023-04-22 14:54:58,528 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/3665561e076f4db790789078bf98e686 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686 2023-04-22 14:54:58,533 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686, entries=150, sequenceid=480, filesize=11.9 K 2023-04-22 14:54:58,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/90e9fbc2771c44a08d476fc74a73fefc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc 2023-04-22 14:54:58,537 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc, entries=150, sequenceid=480, filesize=11.9 K 2023-04-22 14:54:58,538 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06 2023-04-22 14:54:58,541 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06, entries=150, sequenceid=480, filesize=11.9 K 2023-04-22 14:54:58,542 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 5adf52822e4e313e73f3d2c863d10c45 in 47ms, sequenceid=480, compaction requested=true 2023-04-22 14:54:58,542 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,542 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,544 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47eaf36b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,548 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,548 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,548 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,549 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,552 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,552 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,553 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,553 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,553 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,554 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,555 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,558 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,558 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,559 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,559 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,659 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,663 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,663 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,664 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,664 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,665 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,665 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,667 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,667 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,667 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,667 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,669 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,669 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,670 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,670 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a9eda5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,671 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,672 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,674 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,674 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,677 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,677 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,677 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,677 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,677 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,681 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,683 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,683 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,684 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,684 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,684 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,784 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,787 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,796 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,796 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,798 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,798 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,798 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,799 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,800 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,800 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,800 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3cf813[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,802 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,802 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,803 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,803 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,805 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,807 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,807 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,809 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,809 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,809 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,813 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,814 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,814 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,814 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,914 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:58,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:58,918 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:58,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:58,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:58,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,920 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,920 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:58,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:58,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:58,921 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:58,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:58,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:58,923 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:58,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a4adf1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:58,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:58,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,927 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:58,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:58,927 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,927 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:58,928 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:58,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:58,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:58,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:58,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:58,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:58,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:58,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:58,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:58,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,937 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,937 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:58,937 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:58,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:58,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:58,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:58,938 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:58,938 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:58,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,038 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:59,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:59,042 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:59,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:59,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:59,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,044 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:59,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:59,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:59,046 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,046 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:59,047 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:59,047 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:59,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29457005[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:59,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:59,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:59,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:59,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:59,055 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:59,056 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:59,056 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:59,056 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:59,056 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:59,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,059 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:59,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:59,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:59,063 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:59,063 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:59,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,163 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:59,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:59,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:59,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:59,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:59,166 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:59,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:59,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:59,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,168 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,168 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,169 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,169 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:59,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:59,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:59,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:59,170 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,170 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:59,170 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:54:59,170 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:59,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:59,177 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:59,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6644d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:59,177 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:59,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:59,179 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:59,182 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:59,182 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:59,182 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:59,182 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:59,182 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:59,184 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,184 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,184 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,186 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:59,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:59,189 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,189 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:59,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,190 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:59,190 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:54:59,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:54:59,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:54:59,290 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:59,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:59,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:59,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:54:59,300 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:54:59,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:54:59,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:54:59,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,301 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,301 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:54:59,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:54:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:54:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:54:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:59,303 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,303 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. started... 2023-04-22 14:54:59,303 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. as already flushing 2023-04-22 14:54:59,304 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:54:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:54:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:54:59,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,305 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:54:59,307 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3561ebe8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:59,307 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:54:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:54:59,309 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,309 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:54:59,312 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,312 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:54:59,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:54:59,314 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:54:59,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:59,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:59,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:54:59,314 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:59,316 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,316 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,316 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:59,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:59,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:54:59,318 DEBUG [Thread-1471] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:51853 2023-04-22 14:54:59,319 DEBUG [Thread-1471] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,319 DEBUG [Thread-1460] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:51853 2023-04-22 14:54:59,319 DEBUG [Thread-1460] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,320 DEBUG [Thread-1473] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:51853 2023-04-22 14:54:59,320 DEBUG [Thread-1473] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,321 DEBUG [Thread-1477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:51853 2023-04-22 14:54:59,321 DEBUG [Thread-1479] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:51853 2023-04-22 14:54:59,321 DEBUG [Thread-1477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,321 DEBUG [Thread-1479] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,323 DEBUG [Thread-1475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:51853 2023-04-22 14:54:59,323 DEBUG [Thread-1475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:59,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:59,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:59,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:54:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:59,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:59,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:54:59,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:59,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:54:59,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:54:59,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:54:59,324 DEBUG [Thread-1470] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:59,324 DEBUG [Thread-1470] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:59,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:59,335 DEBUG [Thread-1462] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:51853 2023-04-22 14:54:59,335 DEBUG [Thread-1462] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:54:59,424 DEBUG [Thread-1470] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:59,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:59,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/af2efc855bc641e9889acb66414bdaef 2023-04-22 14:55:00,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/96d181aec25d44caa4ef350ad4ded83f 2023-04-22 14:55:00,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/00f30537994945d8a7a435bca2c1c082 2023-04-22 14:55:00,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/af2efc855bc641e9889acb66414bdaef as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef 2023-04-22 14:55:00,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef, entries=100, sequenceid=495, filesize=9.5 K 2023-04-22 14:55:00,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/96d181aec25d44caa4ef350ad4ded83f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f 2023-04-22 14:55:00,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f, entries=100, sequenceid=495, filesize=9.5 K 2023-04-22 14:55:00,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/00f30537994945d8a7a435bca2c1c082 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082 2023-04-22 14:55:00,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082, entries=100, sequenceid=495, filesize=9.5 K 2023-04-22 14:55:00,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=100.63 KB/103050 for 5adf52822e4e313e73f3d2c863d10c45 in 1255ms, sequenceid=495, compaction requested=true 2023-04-22 14:55:00,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:00,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:00,536 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:00,536 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:00,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:00,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:00,537 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:00,537 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:00,537 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/A is initiating minor compaction (all files) 2023-04-22 14:55:00,537 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/B is initiating minor compaction (all files) 2023-04-22 14:55:00,537 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/A in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:00,537 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/B in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:00,537 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/36ac3ab86273401e9691aeb672df743f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=46.2 K 2023-04-22 14:55:00,537 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/4165a58171254746a4660536c4920e20, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=46.2 K 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 36ac3ab86273401e9691aeb672df743f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4165a58171254746a4660536c4920e20, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting dbf1a3668eb74306859de7f99cc3a1ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682175297152 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 27a82009e1024903b38cf50fcb3b8f64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682175297152 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3665561e076f4db790789078bf98e686, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682175297210 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 90e9fbc2771c44a08d476fc74a73fefc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682175297210 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting af2efc855bc641e9889acb66414bdaef, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682175299271 2023-04-22 14:55:00,538 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 96d181aec25d44caa4ef350ad4ded83f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682175299271 2023-04-22 14:55:00,545 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#A#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:00,545 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#B#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:00,953 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/7aa7160b260c4dc39d45a171420e985b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/7aa7160b260c4dc39d45a171420e985b 2023-04-22 14:55:00,953 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/dc18b62e17944627a8c14d6df8b9e2b2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dc18b62e17944627a8c14d6df8b9e2b2 2023-04-22 14:55:00,957 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/B of 5adf52822e4e313e73f3d2c863d10c45 into 7aa7160b260c4dc39d45a171420e985b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:00,957 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/A of 5adf52822e4e313e73f3d2c863d10c45 into dc18b62e17944627a8c14d6df8b9e2b2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:00,957 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:00,957 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:00,957 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/B, priority=12, startTime=1682175300536; duration=0sec 2023-04-22 14:55:00,957 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/A, priority=12, startTime=1682175300536; duration=0sec 2023-04-22 14:55:00,957 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:00,957 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:00,957 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:00,958 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:00,958 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 5adf52822e4e313e73f3d2c863d10c45/C is initiating minor compaction (all files) 2023-04-22 14:55:00,958 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5adf52822e4e313e73f3d2c863d10c45/C in TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:00,958 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/af3e7f1bdddc44edb784932ccba2ce82, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp, totalSize=46.2 K 2023-04-22 14:55:00,959 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting af3e7f1bdddc44edb784932ccba2ce82, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682175296475 2023-04-22 14:55:00,959 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b2b8f43b78d64376a7ef1e07d15c0c63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682175297152 2023-04-22 14:55:00,959 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8a33ef1f7f6a4a7d8eaeb0d951f54c06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682175297210 2023-04-22 14:55:00,959 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 00f30537994945d8a7a435bca2c1c082, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682175299271 2023-04-22 14:55:00,966 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 5adf52822e4e313e73f3d2c863d10c45#C#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:01,374 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/9c5d0edacea24fcba34cec42dea7399f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c5d0edacea24fcba34cec42dea7399f 2023-04-22 14:55:01,377 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5adf52822e4e313e73f3d2c863d10c45/C of 5adf52822e4e313e73f3d2c863d10c45 into 9c5d0edacea24fcba34cec42dea7399f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:01,377 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:01,378 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45., storeName=5adf52822e4e313e73f3d2c863d10c45/C, priority=12, startTime=1682175300536; duration=0sec 2023-04-22 14:55:01,378 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:05,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:55:05,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:55:05,477 DEBUG [Thread-1468] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:51853 2023-04-22 14:55:05,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:55:05,477 DEBUG [Thread-1468] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:05,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:05,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:55:05,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:05,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:55:05,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:05,514 DEBUG [Thread-1466] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:51853 2023-04-22 14:55:05,514 DEBUG [Thread-1466] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:05,522 DEBUG [Thread-1464] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:51853 2023-04-22 14:55:05,522 DEBUG [Thread-1464] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5782 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5609 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5606 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5780 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 5604 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:55:05,522 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:55:05,522 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:51853 2023-04-22 14:55:05,523 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:05,523 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:55:05,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:55:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:05,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:05,526 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175305526"}]},"ts":"1682175305526"} 2023-04-22 14:55:05,527 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:55:05,530 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:55:05,531 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, UNASSIGN}] 2023-04-22 14:55:05,532 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, UNASSIGN 2023-04-22 14:55:05,533 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5adf52822e4e313e73f3d2c863d10c45, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:05,533 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175305533"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175305533"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175305533"}]},"ts":"1682175305533"} 2023-04-22 14:55:05,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:05,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:05,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:55:05,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5adf52822e4e313e73f3d2c863d10c45, disabling compactions & flushes 2023-04-22 14:55:05,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:05,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:05,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/546832199047444e87e2514f594d0e73 2023-04-22 14:55:06,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:06,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/9ee6bd5f92c540569d4c1ec5726b2ad6 2023-04-22 14:55:06,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:06,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/46cc935febdc4ec3898ad202046376d5 2023-04-22 14:55:06,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/546832199047444e87e2514f594d0e73 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/546832199047444e87e2514f594d0e73 2023-04-22 14:55:06,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/546832199047444e87e2514f594d0e73, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 14:55:06,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/9ee6bd5f92c540569d4c1ec5726b2ad6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9ee6bd5f92c540569d4c1ec5726b2ad6 2023-04-22 14:55:06,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9ee6bd5f92c540569d4c1ec5726b2ad6, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 14:55:06,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/46cc935febdc4ec3898ad202046376d5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/46cc935febdc4ec3898ad202046376d5 2023-04-22 14:55:06,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/46cc935febdc4ec3898ad202046376d5, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 14:55:06,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for 5adf52822e4e313e73f3d2c863d10c45 in 1242ms, sequenceid=517, compaction requested=false 2023-04-22 14:55:06,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:06,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. after waiting 0 ms 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:06,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5adf52822e4e313e73f3d2c863d10c45 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=A 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=B 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:06,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5adf52822e4e313e73f3d2c863d10c45, store=C 2023-04-22 14:55:06,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:07,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/51b3a285a2fb427e91034246eef42641 2023-04-22 14:55:07,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/db015246e3404fcfaf5e87545dd6bada 2023-04-22 14:55:07,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:07,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/c7e27fa92f334e04908f54367f5b2e2b 2023-04-22 14:55:07,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/A/51b3a285a2fb427e91034246eef42641 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/51b3a285a2fb427e91034246eef42641 2023-04-22 14:55:07,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/51b3a285a2fb427e91034246eef42641, entries=50, sequenceid=522, filesize=7.1 K 2023-04-22 14:55:07,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/B/db015246e3404fcfaf5e87545dd6bada as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/db015246e3404fcfaf5e87545dd6bada 2023-04-22 14:55:07,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/db015246e3404fcfaf5e87545dd6bada, entries=50, sequenceid=522, filesize=7.1 K 2023-04-22 14:55:07,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/.tmp/C/c7e27fa92f334e04908f54367f5b2e2b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c7e27fa92f334e04908f54367f5b2e2b 2023-04-22 14:55:07,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c7e27fa92f334e04908f54367f5b2e2b, entries=50, sequenceid=522, filesize=7.1 K 2023-04-22 14:55:07,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 5adf52822e4e313e73f3d2c863d10c45 in 1240ms, sequenceid=522, compaction requested=true 2023-04-22 14:55:07,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/895ece500dde496481a4738f036e0e81, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/c57b3e510c6e495f98672e34e9ef6510, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/585e5e6d9f1d46a39cefd14df3205c2b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/b14a2bf424084ec3bb39c93efd851ee5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/71df6675490248318e3bb213f914141b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fee573e81e75450c95ce5e3ee7c50380, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/36ac3ab86273401e9691aeb672df743f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef] to archive 2023-04-22 14:55:07,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:07,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3c866bea86a8401ebc431c1252862998 2023-04-22 14:55:07,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/a3ffb7b7f10b42efb73051eaf52f0ce6 2023-04-22 14:55:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/895ece500dde496481a4738f036e0e81 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/895ece500dde496481a4738f036e0e81 2023-04-22 14:55:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/897f1753f2d6467488bc3dc0c85ec9f6 2023-04-22 14:55:07,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fdbb871837594d07b22afc78dfde0ec6 2023-04-22 14:55:07,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/52aa96f9f44a4db098ecc3804230cdf2 2023-04-22 14:55:07,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1ff5be1adac2493f9dac276180802444 2023-04-22 14:55:07,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/cceb9c728a974c648e252f33f473b1d9 2023-04-22 14:55:07,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/27975ed489ba4f49a701b8e0c0a5285f 2023-04-22 14:55:07,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/2ad179cfcc524550b0254d8fec393e57 2023-04-22 14:55:07,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/0e4add539bde473994ac09ecc8c5490a 2023-04-22 14:55:07,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/339fd7a5725842d386faf98c888867f6 2023-04-22 14:55:07,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/c57b3e510c6e495f98672e34e9ef6510 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/c57b3e510c6e495f98672e34e9ef6510 2023-04-22 14:55:07,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/077be341a0b549c084438eab9c8c8c99 2023-04-22 14:55:07,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/585e5e6d9f1d46a39cefd14df3205c2b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/585e5e6d9f1d46a39cefd14df3205c2b 2023-04-22 14:55:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/7c35eae04e774356ba9ffa946c201239 2023-04-22 14:55:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/aac332072a8e4831a8622df4e1d1659d 2023-04-22 14:55:07,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/70376881438746edbc1e557f2c496847 2023-04-22 14:55:08,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/8d0a72d4606a498197038f9f2a7874d3 2023-04-22 14:55:08,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/b14a2bf424084ec3bb39c93efd851ee5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/b14a2bf424084ec3bb39c93efd851ee5 2023-04-22 14:55:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/90d7db89036f48eab88454f6229726bb 2023-04-22 14:55:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/08c4db1256d24aceb69f149e41680abe 2023-04-22 14:55:08,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/63cd01aed7f3405891c4b0dd39813559 2023-04-22 14:55:08,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/71df6675490248318e3bb213f914141b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/71df6675490248318e3bb213f914141b 2023-04-22 14:55:08,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/1efe24d6329f4430b7608318f52240b8 2023-04-22 14:55:08,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/ee1e2f9384264382a156e609252b0395 2023-04-22 14:55:08,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fee573e81e75450c95ce5e3ee7c50380 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/fee573e81e75450c95ce5e3ee7c50380 2023-04-22 14:55:08,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/395eb36ba05f4567bdeb5e7f8763ecdf 2023-04-22 14:55:08,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/564010b25c2e4d928d03200a6cfd86f1 2023-04-22 14:55:08,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/d3eb0d7426514735a262d5e58297ca39 2023-04-22 14:55:08,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/36ac3ab86273401e9691aeb672df743f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/36ac3ab86273401e9691aeb672df743f 2023-04-22 14:55:08,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dbf1a3668eb74306859de7f99cc3a1ad 2023-04-22 14:55:08,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/3665561e076f4db790789078bf98e686 2023-04-22 14:55:08,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/af2efc855bc641e9889acb66414bdaef 2023-04-22 14:55:08,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9362779feaf2416fa15ac106d2f4d394, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5cfc594d644645a49f1dc027ad5ea73d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/299f02d7dcc1474cb46b3042a418c360, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/844a35afa71e4a3fb4925ba7dc9f6b66, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/513a8d9fc0be4b4fbb4bdf0893ccc557, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5ca61b45e6dd412da65413ab238fcec7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/4165a58171254746a4660536c4920e20, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f] to archive 2023-04-22 14:55:08,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:08,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f8a8c6fa60e34153978f3f7fb37b0dbd 2023-04-22 14:55:08,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c24176d292c44257befc6e56f0f62f36 2023-04-22 14:55:08,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9362779feaf2416fa15ac106d2f4d394 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9362779feaf2416fa15ac106d2f4d394 2023-04-22 14:55:08,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/b17a5b69eebd4c2dab17f882854b5310 2023-04-22 14:55:08,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/8da379461a514325b75b5a0d303e9004 2023-04-22 14:55:08,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/281b28f0e9024a11b12320812e2f0203 2023-04-22 14:55:08,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c292f79ddb8f4e4096f76740334e420e 2023-04-22 14:55:08,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/777cdaf8c8264efeaf2f8fd3426ec685 2023-04-22 14:55:08,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/1124cb30531e4504aa2992dffdf9cbbd 2023-04-22 14:55:08,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ac8808c465b94f1abb183a65cfa15bc6 2023-04-22 14:55:08,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/75999ba799e741f997cdba6847bffb5d 2023-04-22 14:55:08,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5cfc594d644645a49f1dc027ad5ea73d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5cfc594d644645a49f1dc027ad5ea73d 2023-04-22 14:55:08,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/930bd97959ca4ed7bcf755b7b0d2aeb0 2023-04-22 14:55:08,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/662037dbf62b423eac0e34a0b9ef946a 2023-04-22 14:55:08,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/299f02d7dcc1474cb46b3042a418c360 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/299f02d7dcc1474cb46b3042a418c360 2023-04-22 14:55:08,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/ae084cb176d34845b2ec9a49d5ee3c6a 2023-04-22 14:55:08,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/abf37bdae4c443adbf506be7de4bd8ea 2023-04-22 14:55:08,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/fe96b5dcef844679b76a5cbdf659130d 2023-04-22 14:55:08,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/844a35afa71e4a3fb4925ba7dc9f6b66 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/844a35afa71e4a3fb4925ba7dc9f6b66 2023-04-22 14:55:08,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/a659a7a242004536afc92156b341aa17 2023-04-22 14:55:08,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5aaf7b301b784eb9a921c2f8d200d0e3 2023-04-22 14:55:08,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/f5fad8a7e1e44f05b4bed2fbac9a08f9 2023-04-22 14:55:08,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/05eb5816814b464c9377b6fec6b829b5 2023-04-22 14:55:08,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/513a8d9fc0be4b4fbb4bdf0893ccc557 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/513a8d9fc0be4b4fbb4bdf0893ccc557 2023-04-22 14:55:08,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/35ad69c614784a869ca0545e7fa070c0 2023-04-22 14:55:08,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/e4c0178cf34f4e368ef89414b89cb66b 2023-04-22 14:55:08,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5ca61b45e6dd412da65413ab238fcec7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/5ca61b45e6dd412da65413ab238fcec7 2023-04-22 14:55:08,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/610bb7372b3d4891a78c989c1f618ef7 2023-04-22 14:55:08,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/65e0e52d785f415eb2cd3c6cf451b101 2023-04-22 14:55:08,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/4165a58171254746a4660536c4920e20 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/4165a58171254746a4660536c4920e20 2023-04-22 14:55:08,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/c27eb75de84846d3bde015ee11a314e0 2023-04-22 14:55:08,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/27a82009e1024903b38cf50fcb3b8f64 2023-04-22 14:55:08,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/90e9fbc2771c44a08d476fc74a73fefc 2023-04-22 14:55:08,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/96d181aec25d44caa4ef350ad4ded83f 2023-04-22 14:55:08,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ecffb1ad887449598cedb66b681abc85, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8f99e9669fd24ba9a8688233a7d0c8ad, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/53d430d0ee51492a9983211da81ee7fa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4a1baa18600f4c65b9db63563d227a51, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a1466d8e658249cb9e44db5f97b0d0c2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8ebb19f0aa164d1c97704508230618de, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/af3e7f1bdddc44edb784932ccba2ce82, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082] to archive 2023-04-22 14:55:08,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:08,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c717f8ccee4497e9d0854bf2a877e00 2023-04-22 14:55:08,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c46e94bbcb634985a94568c7a1f99f5b 2023-04-22 14:55:08,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ecffb1ad887449598cedb66b681abc85 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ecffb1ad887449598cedb66b681abc85 2023-04-22 14:55:08,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a81e8a64035744c8adbb7e32c7bbc2e7 2023-04-22 14:55:08,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/95c6d5bf04c846209fda40b066690c0a 2023-04-22 14:55:08,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a4b7b8bd71ad48809d26f07d359e1af5 2023-04-22 14:55:08,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fdc383869bdb443db74b2917ca4e4ead 2023-04-22 14:55:08,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/fa6d311a09ee4841b1b881e59ec3fefd 2023-04-22 14:55:08,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/e5cf52230c7b4944af89b9a03529f8f8 2023-04-22 14:55:08,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/cc2a32728a0c49d3a3aef4cef7d04833 2023-04-22 14:55:08,057 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/0c5eca968f29497b98caffc6d8b4f194 2023-04-22 14:55:08,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8f99e9669fd24ba9a8688233a7d0c8ad to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8f99e9669fd24ba9a8688233a7d0c8ad 2023-04-22 14:55:08,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a5d5e53020b74f4a861bde2fe3465b88 2023-04-22 14:55:08,059 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4407633b8be34557a53a43784ae1e1ab 2023-04-22 14:55:08,060 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/53d430d0ee51492a9983211da81ee7fa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/53d430d0ee51492a9983211da81ee7fa 2023-04-22 14:55:08,061 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/50e57b447d52447188c312776383fdc7 2023-04-22 14:55:08,062 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/1a170a64f97b4af096702691912f1445 2023-04-22 14:55:08,063 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/edae34d79a874c4ba8749aae8a6c084f 2023-04-22 14:55:08,064 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4a1baa18600f4c65b9db63563d227a51 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/4a1baa18600f4c65b9db63563d227a51 2023-04-22 14:55:08,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/89cc6f92b82b4cfa92bdb899ca060ee2 2023-04-22 14:55:08,066 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8d2f676782254f60888f9c48011946cd 2023-04-22 14:55:08,067 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/03d2b22b059a449c972d0c25ba35c647 2023-04-22 14:55:08,068 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d9102e7f1b0e46ec80dfb8337c351f39 2023-04-22 14:55:08,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a1466d8e658249cb9e44db5f97b0d0c2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a1466d8e658249cb9e44db5f97b0d0c2 2023-04-22 14:55:08,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/d5a75dcbcc3c49dc82c54979261705a2 2023-04-22 14:55:08,070 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/ec400c71d1474c86ae21baac4fec55fd 2023-04-22 14:55:08,071 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8ebb19f0aa164d1c97704508230618de to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8ebb19f0aa164d1c97704508230618de 2023-04-22 14:55:08,072 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/04c3f52d0852400b8d3824452f48893e 2023-04-22 14:55:08,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/25345795a92d42009394c564ef46dc37 2023-04-22 14:55:08,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/af3e7f1bdddc44edb784932ccba2ce82 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/af3e7f1bdddc44edb784932ccba2ce82 2023-04-22 14:55:08,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/a30c7f41c1034d0d9682c5efdfbb81ae 2023-04-22 14:55:08,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/b2b8f43b78d64376a7ef1e07d15c0c63 2023-04-22 14:55:08,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/8a33ef1f7f6a4a7d8eaeb0d951f54c06 2023-04-22 14:55:08,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/00f30537994945d8a7a435bca2c1c082 2023-04-22 14:55:08,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/recovered.edits/525.seqid, newMaxSeqId=525, maxSeqId=1 2023-04-22 14:55:08,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45. 2023-04-22 14:55:08,081 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5adf52822e4e313e73f3d2c863d10c45: 2023-04-22 14:55:08,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:55:08,082 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5adf52822e4e313e73f3d2c863d10c45, regionState=CLOSED 2023-04-22 14:55:08,082 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175308082"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175308082"}]},"ts":"1682175308082"} 2023-04-22 14:55:08,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-22 14:55:08,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 5adf52822e4e313e73f3d2c863d10c45, server=jenkins-hbase4.apache.org,32935,1682175218140 in 2.5490 sec 2023-04-22 14:55:08,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-22 14:55:08,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5adf52822e4e313e73f3d2c863d10c45, UNASSIGN in 2.5530 sec 2023-04-22 14:55:08,086 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175308086"}]},"ts":"1682175308086"} 2023-04-22 14:55:08,087 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:55:08,088 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:55:08,089 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5660 sec 2023-04-22 14:55:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-22 14:55:09,632 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-22 14:55:09,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:55:09,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,635 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-22 14:55:09,635 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,639 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:55:09,641 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/recovered.edits] 2023-04-22 14:55:09,644 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/51b3a285a2fb427e91034246eef42641 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/51b3a285a2fb427e91034246eef42641 2023-04-22 14:55:09,645 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/546832199047444e87e2514f594d0e73 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/546832199047444e87e2514f594d0e73 2023-04-22 14:55:09,646 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dc18b62e17944627a8c14d6df8b9e2b2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/A/dc18b62e17944627a8c14d6df8b9e2b2 2023-04-22 14:55:09,648 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/7aa7160b260c4dc39d45a171420e985b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/7aa7160b260c4dc39d45a171420e985b 2023-04-22 14:55:09,650 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9ee6bd5f92c540569d4c1ec5726b2ad6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/9ee6bd5f92c540569d4c1ec5726b2ad6 2023-04-22 14:55:09,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/db015246e3404fcfaf5e87545dd6bada to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/B/db015246e3404fcfaf5e87545dd6bada 2023-04-22 14:55:09,653 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/46cc935febdc4ec3898ad202046376d5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/46cc935febdc4ec3898ad202046376d5 2023-04-22 14:55:09,655 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c5d0edacea24fcba34cec42dea7399f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/9c5d0edacea24fcba34cec42dea7399f 2023-04-22 14:55:09,656 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c7e27fa92f334e04908f54367f5b2e2b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/C/c7e27fa92f334e04908f54367f5b2e2b 2023-04-22 14:55:09,658 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/recovered.edits/525.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45/recovered.edits/525.seqid 2023-04-22 14:55:09,659 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/5adf52822e4e313e73f3d2c863d10c45 2023-04-22 14:55:09,659 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:55:09,661 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,662 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:55:09,664 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:55:09,665 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,665 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:55:09,665 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175309665"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:09,666 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:55:09,666 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5adf52822e4e313e73f3d2c863d10c45, NAME => 'TestAcidGuarantees,,1682175277142.5adf52822e4e313e73f3d2c863d10c45.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:55:09,666 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:55:09,667 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175309666"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:09,668 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:55:09,669 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,670 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 38 msec 2023-04-22 14:55:09,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-22 14:55:09,736 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-22 14:55:09,748 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225 (was 233), OpenFileDescriptor=548 (was 542) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=349 (was 415), ProcessCount=172 (was 172), AvailableMemoryMB=12440 (was 12472) 2023-04-22 14:55:09,759 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=225, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=349, ProcessCount=172, AvailableMemoryMB=12439 2023-04-22 14:55:09,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:55:09,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:55:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:09,764 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:09,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-22 14:55:09,765 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:09,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:09,766 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:09,766 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e empty. 2023-04-22 14:55:09,767 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:09,767 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:55:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:10,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:10,174 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:10,176 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0c924aecfddd8b7ecdab058989640d2e, NAME => 'TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:55:10,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0c924aecfddd8b7ecdab058989640d2e, disabling compactions & flushes 2023-04-22 14:55:10,582 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. after waiting 0 ms 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,582 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:10,584 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:10,585 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175310585"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175310585"}]},"ts":"1682175310585"} 2023-04-22 14:55:10,586 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:10,586 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:10,587 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175310587"}]},"ts":"1682175310587"} 2023-04-22 14:55:10,587 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:55:10,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, ASSIGN}] 2023-04-22 14:55:10,591 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, ASSIGN 2023-04-22 14:55:10,592 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:55:10,743 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:10,743 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175310743"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175310743"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175310743"}]},"ts":"1682175310743"} 2023-04-22 14:55:10,745 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:10,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:10,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0c924aecfddd8b7ecdab058989640d2e, NAME => 'TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:10,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:10,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,901 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,902 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A 2023-04-22 14:55:10,902 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A 2023-04-22 14:55:10,902 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:10,903 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName A 2023-04-22 14:55:10,903 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:10,903 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,904 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B 2023-04-22 14:55:10,904 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B 2023-04-22 14:55:10,904 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:10,905 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName B 2023-04-22 14:55:10,905 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:10,905 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,906 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C 2023-04-22 14:55:10,906 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C 2023-04-22 14:55:10,906 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:10,906 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName C 2023-04-22 14:55:10,907 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:10,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:55:10,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:10,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:10,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0c924aecfddd8b7ecdab058989640d2e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73781920, jitterRate=0.09943628311157227}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:55:10,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:10,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., pid=37, masterSystemTime=1682175310896 2023-04-22 14:55:10,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:10,914 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:10,915 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175310914"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175310914"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175310914"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175310914"}]},"ts":"1682175310914"} 2023-04-22 14:55:10,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-22 14:55:10,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 in 170 msec 2023-04-22 14:55:10,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-22 14:55:10,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, ASSIGN in 327 msec 2023-04-22 14:55:10,919 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:10,919 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175310919"}]},"ts":"1682175310919"} 2023-04-22 14:55:10,920 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:55:10,921 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:10,922 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-22 14:55:11,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:55:11,870 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-22 14:55:11,870 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:11,876 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:11,877 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:11,879 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:11,880 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:55:11,881 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:55:11,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:55:11,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:55:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:12,294 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:55:12,295 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:12,295 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:55:12,297 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 14:55:12,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, REOPEN/MOVE}] 2023-04-22 14:55:12,300 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, REOPEN/MOVE 2023-04-22 14:55:12,301 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,301 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175312301"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175312301"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175312301"}]},"ts":"1682175312301"} 2023-04-22 14:55:12,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:12,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0c924aecfddd8b7ecdab058989640d2e, disabling compactions & flushes 2023-04-22 14:55:12,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. after waiting 0 ms 2023-04-22 14:55:12,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 14:55:12,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:12,458 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 0c924aecfddd8b7ecdab058989640d2e to self. 2023-04-22 14:55:12,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,460 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=CLOSED 2023-04-22 14:55:12,460 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175312460"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175312460"}]},"ts":"1682175312460"} 2023-04-22 14:55:12,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-22 14:55:12,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 in 159 msec 2023-04-22 14:55:12,463 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=true 2023-04-22 14:55:12,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,613 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175312613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175312613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175312613"}]},"ts":"1682175312613"} 2023-04-22 14:55:12,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:12,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0c924aecfddd8b7ecdab058989640d2e, NAME => 'TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,771 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,772 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A 2023-04-22 14:55:12,772 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A 2023-04-22 14:55:12,773 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:12,773 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName A 2023-04-22 14:55:12,774 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:12,774 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,775 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B 2023-04-22 14:55:12,775 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B 2023-04-22 14:55:12,775 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:12,775 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName B 2023-04-22 14:55:12,776 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:12,776 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,776 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C 2023-04-22 14:55:12,776 DEBUG [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C 2023-04-22 14:55:12,777 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:12,777 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c924aecfddd8b7ecdab058989640d2e columnFamilyName C 2023-04-22 14:55:12,777 INFO [StoreOpener-0c924aecfddd8b7ecdab058989640d2e-1] regionserver.HStore(310): Store=0c924aecfddd8b7ecdab058989640d2e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:12,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:55:12,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0c924aecfddd8b7ecdab058989640d2e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70376091, jitterRate=0.048685476183891296}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:55:12,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:12,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., pid=42, masterSystemTime=1682175312766 2023-04-22 14:55:12,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175312784"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175312784"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175312784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175312784"}]},"ts":"1682175312784"} 2023-04-22 14:55:12,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-22 14:55:12,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 in 170 msec 2023-04-22 14:55:12,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-22 14:55:12,787 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, REOPEN/MOVE in 487 msec 2023-04-22 14:55:12,788 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-22 14:55:12,788 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-22 14:55:12,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-22 14:55:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-22 14:55:12,791 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,796 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,796 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,799 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,800 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,804 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,804 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,809 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,809 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,812 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,812 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,816 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,816 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,820 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,820 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,826 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,827 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,830 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,830 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:12,833 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:12,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:12,844 DEBUG [hconnection-0x3b1683db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,844 DEBUG [hconnection-0x342d6288-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,845 DEBUG [hconnection-0x27880090-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,846 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,846 DEBUG [hconnection-0x6af5646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,846 DEBUG [hconnection-0x7e1e2733-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,845 DEBUG [hconnection-0x3cb255ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,846 DEBUG [hconnection-0x14e1631d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:12,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:12,846 DEBUG [hconnection-0x1376e3fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,847 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,848 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:12,848 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:12,849 DEBUG [hconnection-0x1adeda5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,849 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:12,850 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:12,850 DEBUG [hconnection-0x4e41bd5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:12,851 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,850 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50334, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,851 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,851 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,853 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,853 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:12,852 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:12,856 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:12,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:12,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:12,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:12,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:12,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:12,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:12,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:12,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:12,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:12,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:12,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:12,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:12,868 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:12,871 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:12,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175372909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:12,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175372912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:12,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175372913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175372916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:12,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175372917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175373014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175373017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175373018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175373022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175373022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175373219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175373223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175373223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175373226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175373227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,294 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b508e11eb6784c1594ee82a6a48d185d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b508e11eb6784c1594ee82a6a48d185d 2023-04-22 14:55:13,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175373523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175373527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175373528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175373529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:13,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175373533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:13,612 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:55:13,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/5dff9f1815b84cee8c82469e92c242c7 2023-04-22 14:55:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175374030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:14,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175374032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175374035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:14,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175374035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175374038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,112 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/ddf9dc7eab53417ea88681d15af4e298 2023-04-22 14:55:14,523 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/17bd5f36b610440993f9e7caa1297b12 2023-04-22 14:55:14,527 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/5dff9f1815b84cee8c82469e92c242c7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7 2023-04-22 14:55:14,531 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7, entries=200, sequenceid=18, filesize=31.9 K 2023-04-22 14:55:14,532 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/ddf9dc7eab53417ea88681d15af4e298 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298 2023-04-22 14:55:14,536 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298, entries=150, sequenceid=18, filesize=11.6 K 2023-04-22 14:55:14,536 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/17bd5f36b610440993f9e7caa1297b12 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12 2023-04-22 14:55:14,540 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12, entries=150, sequenceid=18, filesize=11.6 K 2023-04-22 14:55:14,541 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 0c924aecfddd8b7ecdab058989640d2e in 1673ms, sequenceid=18, compaction requested=false 2023-04-22 14:55:14,541 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 14:55:14,542 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:14,542 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:14,544 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:14,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:14,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b61390[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:14,546 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:14,546 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,547 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:14,547 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:14,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:14,548 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:14,551 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:14,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:14,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:14,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:14,551 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:14,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:14,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:14,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,555 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,556 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:14,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,559 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,560 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:14,560 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:14,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:14,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:14,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:14,560 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:14,560 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:14,660 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:14,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:14,664 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:14,664 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:14,664 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:14,664 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:14,665 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,665 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:14,666 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:14,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:14,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:14,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:14,666 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:14,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:14,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:14,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:14,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:14,668 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:14,668 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:14,668 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:14,686 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c91036c3452d459d90fac685db571190 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c91036c3452d459d90fac685db571190 2023-04-22 14:55:15,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:15,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:15,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175375045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175375045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175375046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175375050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175375051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,094 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/86b484f4f4d44286a5df2d8339c91238 2023-04-22 14:55:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175375152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175375152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175375154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175375154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175375358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175375358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175375359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175375359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,504 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/a88771ae416347028843cdeacc52f27a 2023-04-22 14:55:15,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175375663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175375664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175375664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175375665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,918 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ab0857e7b321482696a220c50711604a 2023-04-22 14:55:15,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/86b484f4f4d44286a5df2d8339c91238 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238 2023-04-22 14:55:15,926 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238, entries=150, sequenceid=37, filesize=25.1 K 2023-04-22 14:55:15,927 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/a88771ae416347028843cdeacc52f27a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a 2023-04-22 14:55:15,931 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a, entries=150, sequenceid=37, filesize=11.6 K 2023-04-22 14:55:15,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ab0857e7b321482696a220c50711604a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a 2023-04-22 14:55:15,938 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a, entries=150, sequenceid=37, filesize=11.6 K 2023-04-22 14:55:15,939 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0c924aecfddd8b7ecdab058989640d2e in 1271ms, sequenceid=37, compaction requested=false 2023-04-22 14:55:15,939 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:15,939 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:15,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:15,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:15,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,941 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:15,941 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:15,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:15,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:15,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf19bfb[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:15,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:15,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,946 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:15,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:15,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:15,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:15,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:15,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:15,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:15,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:15,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:15,951 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:15,951 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:15,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:15,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:15,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:15,955 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,956 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:15,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:15,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:15,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:15,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:15,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:15,958 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:15,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,959 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:15,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:15,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:15,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,059 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:16,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:16,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:16,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:16,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:16,062 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:16,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:16,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:16,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:16,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:16,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:16,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:16,065 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,065 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:16,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:16,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:16,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:16,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:16,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:16,066 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:16,067 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:16,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:16,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175376217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175376218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175376223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175376224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175376328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175376328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175376330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175376333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,482 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042266d85439d2d44c1f927c0c5f2f5f19e5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042266d85439d2d44c1f927c0c5f2f5f19e5 2023-04-22 14:55:16,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175376534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175376534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175376535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175376540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175376838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175376838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175376841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:16,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175376845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,891 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/8af4f97b513942d2bea94b578f54b131 2023-04-22 14:55:16,902 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/ecd8da92a0a545e6a2c05bc11128026c 2023-04-22 14:55:16,910 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/582c7624a52b4150ae47dc909155644b 2023-04-22 14:55:16,916 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/8af4f97b513942d2bea94b578f54b131 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131 2023-04-22 14:55:16,927 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131, entries=150, sequenceid=50, filesize=25.1 K 2023-04-22 14:55:16,928 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/ecd8da92a0a545e6a2c05bc11128026c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c 2023-04-22 14:55:16,932 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c, entries=150, sequenceid=50, filesize=11.6 K 2023-04-22 14:55:16,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/582c7624a52b4150ae47dc909155644b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b 2023-04-22 14:55:16,941 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b, entries=150, sequenceid=50, filesize=11.6 K 2023-04-22 14:55:16,942 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0c924aecfddd8b7ecdab058989640d2e in 876ms, sequenceid=50, compaction requested=true 2023-04-22 14:55:16,942 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:16,942 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:16,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:16,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:16,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:16,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:16,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64806c4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:16,948 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,948 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:16,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:16,951 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:16,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:16,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:16,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:16,952 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:16,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:16,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:16,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,955 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:16,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:16,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:16,959 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:16,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:16,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:16,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:16,959 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:16,960 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:16,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:17,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:17,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:17,060 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,063 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,066 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,067 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,067 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,076 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccc934e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,082 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,082 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,082 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,082 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,087 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,087 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,087 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,087 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,087 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,088 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,089 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,091 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,091 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,091 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:17,091 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,092 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,092 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175377115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,192 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,196 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,197 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,198 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,199 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,199 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,199 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,199 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,202 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ff3dd4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,205 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,205 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,206 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,206 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,208 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,210 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,210 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,214 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,217 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,218 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,218 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175377226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,318 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,321 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,322 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,322 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,329 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,329 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,329 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,329 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,331 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27644d3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,332 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,333 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,333 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,334 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,334 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,334 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,341 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,341 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,341 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,347 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,347 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,347 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,348 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175377344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175377344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175377348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175377348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175377432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,448 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,451 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,452 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,452 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,452 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,452 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,452 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,453 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,454 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,454 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,455 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,455 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,455 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,455 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,458 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444e1c7f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,461 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,461 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,461 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,462 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,462 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,465 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,467 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,467 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,470 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,470 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,470 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,471 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,472 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,475 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,475 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,475 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,476 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042277129ff3161e4267946b1fd2fa2c324b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277129ff3161e4267946b1fd2fa2c324b 2023-04-22 14:55:17,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,575 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,579 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,580 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,581 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,583 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,583 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,583 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,583 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc0a15b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,586 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,587 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,590 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,592 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,593 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,593 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,595 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,595 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,595 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,596 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,597 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,599 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,600 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,600 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,600 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,600 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,700 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,704 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,709 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,709 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,709 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,709 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,711 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,711 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,711 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,711 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408d3e8a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,713 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,714 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,714 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,714 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,714 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,714 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,714 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,715 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,718 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,718 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,723 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,724 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,727 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,727 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,727 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,727 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,727 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,727 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,727 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,727 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,728 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:17,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175377737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,827 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,831 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,833 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,834 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,834 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,834 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,835 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,835 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,835 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,835 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,837 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,837 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,838 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,838 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@706daa94[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,839 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,840 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,840 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,840 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,844 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,844 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,845 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,845 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,846 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,847 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,855 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,855 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,855 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,855 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:17,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,882 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/3943591c34a1470d83383d2c0af16703 2023-04-22 14:55:17,955 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:17,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:17,958 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:17,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:17,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:17,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,961 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,962 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,962 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:17,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:17,963 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,963 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:17,963 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:17,963 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:17,966 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,966 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:17,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4ed02[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:17,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:17,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:17,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:17,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:17,970 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:17,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:17,973 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:17,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:17,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:17,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:17,973 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:17,974 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,974 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,974 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:17,975 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,976 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:17,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:17,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:17,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:17,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:17,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:17,981 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:17,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:17,981 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,081 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,084 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,085 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,085 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,086 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,088 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,088 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,088 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:18,088 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,090 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,090 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da756be[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,094 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,095 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,095 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,095 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,095 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,097 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,097 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,097 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,097 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,097 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,099 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,099 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,103 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,103 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,203 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,207 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,208 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,209 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,209 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,209 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,211 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,211 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,211 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:18,211 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,212 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74775aee[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,219 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,219 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,219 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,221 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,223 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,224 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,231 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,232 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,232 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:18,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175378246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/08484b3ce1874e2699be6c032e616a84 2023-04-22 14:55:18,332 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,336 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,337 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,337 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,339 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,339 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,339 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:18,339 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,341 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695367fe[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,346 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,349 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,349 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,349 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,349 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,349 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,350 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,350 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,350 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175378350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,355 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,355 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,355 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,356 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:18,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175378354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175378357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:18,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175378361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,456 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,459 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,459 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,459 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,459 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,460 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,462 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,462 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,464 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,464 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,464 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:18,464 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,464 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,466 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2c0b2b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,467 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,469 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,469 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,469 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,472 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,475 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,475 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,475 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,478 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,478 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,478 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,478 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,482 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:18,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,482 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,482 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,582 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,604 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,606 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,616 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,616 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,617 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:18,617 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd9ae76[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,623 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,628 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,628 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,630 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,636 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:18,636 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/9946f61b6c5c4d71853ca2a1b97c5ac8 2023-04-22 14:55:18,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/3943591c34a1470d83383d2c0af16703 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703 2023-04-22 14:55:18,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703, entries=200, sequenceid=70, filesize=31.9 K 2023-04-22 14:55:18,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/08484b3ce1874e2699be6c032e616a84 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84 2023-04-22 14:55:18,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84, entries=150, sequenceid=70, filesize=11.6 K 2023-04-22 14:55:18,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/9946f61b6c5c4d71853ca2a1b97c5ac8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8 2023-04-22 14:55:18,736 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8, entries=150, sequenceid=70, filesize=11.6 K 2023-04-22 14:55:18,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:18,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:18,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:18,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:18,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:18,740 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0c924aecfddd8b7ecdab058989640d2e in 1682ms, sequenceid=70, compaction requested=true 2023-04-22 14:55:18,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:18,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,742 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:18,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:18,743 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:18,743 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:18,743 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:18,744 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 116864 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:18,744 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/A is initiating minor compaction (all files) 2023-04-22 14:55:18,744 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/A in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,745 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=114.1 K 2023-04-22 14:55:18,745 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5dff9f1815b84cee8c82469e92c242c7, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682175312865 2023-04-22 14:55:18,745 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 86b484f4f4d44286a5df2d8339c91238, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682175312908 2023-04-22 14:55:18,746 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8af4f97b513942d2bea94b578f54b131, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682175315044 2023-04-22 14:55:18,746 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3943591c34a1470d83383d2c0af16703, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316215 2023-04-22 14:55:18,744 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:18,747 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/B is initiating minor compaction (all files) 2023-04-22 14:55:18,747 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/B in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,748 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=46.4 K 2023-04-22 14:55:18,748 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ddf9dc7eab53417ea88681d15af4e298, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682175312867 2023-04-22 14:55:18,749 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a88771ae416347028843cdeacc52f27a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682175312908 2023-04-22 14:55:18,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,751 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ecd8da92a0a545e6a2c05bc11128026c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682175315044 2023-04-22 14:55:18,751 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 08484b3ce1874e2699be6c032e616a84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316222 2023-04-22 14:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:18,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,756 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:18,757 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:18,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:18,783 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#B#compaction#306 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:18,806 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#A#compaction#305 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:18,850 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042242cc23a61d5248109d93ee121c9d1479 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042242cc23a61d5248109d93ee121c9d1479 2023-04-22 14:55:18,884 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/0f19c85d0c60455bbc8a88e833965f71 2023-04-22 14:55:18,890 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/83ea88d3c2284282a504c32d200a7724 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/83ea88d3c2284282a504c32d200a7724 2023-04-22 14:55:18,898 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/A of 0c924aecfddd8b7ecdab058989640d2e into 83ea88d3c2284282a504c32d200a7724(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:18,898 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:18,898 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/A, priority=12, startTime=1682175318742; duration=0sec 2023-04-22 14:55:18,898 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:18,898 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:18,900 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:18,900 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/C is initiating minor compaction (all files) 2023-04-22 14:55:18,900 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/C in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,900 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=46.4 K 2023-04-22 14:55:18,900 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 17bd5f36b610440993f9e7caa1297b12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682175312867 2023-04-22 14:55:18,904 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ab0857e7b321482696a220c50711604a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682175312908 2023-04-22 14:55:18,904 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 582c7624a52b4150ae47dc909155644b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682175315044 2023-04-22 14:55:18,905 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9946f61b6c5c4d71853ca2a1b97c5ac8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316222 2023-04-22 14:55:18,906 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/3372c0f8435d450488c52a6a0aa223f5 2023-04-22 14:55:18,919 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:18,933 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/75655502333c42ae9eb4244a96e2b4c9 2023-04-22 14:55:18,936 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/0f19c85d0c60455bbc8a88e833965f71 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71 2023-04-22 14:55:18,940 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71, entries=150, sequenceid=82, filesize=25.1 K 2023-04-22 14:55:18,942 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/3372c0f8435d450488c52a6a0aa223f5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5 2023-04-22 14:55:18,945 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5, entries=150, sequenceid=82, filesize=11.6 K 2023-04-22 14:55:18,951 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/75655502333c42ae9eb4244a96e2b4c9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9 2023-04-22 14:55:18,957 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9, entries=150, sequenceid=82, filesize=11.6 K 2023-04-22 14:55:18,957 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0c924aecfddd8b7ecdab058989640d2e in 201ms, sequenceid=82, compaction requested=false 2023-04-22 14:55:18,957 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:18,958 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:18,960 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b72909[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,962 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:18,962 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:18,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,973 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,977 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:18,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:18,977 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,982 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,985 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:18,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,985 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,985 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,986 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,986 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:18,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,086 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:19,092 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:19,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:19,092 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:19,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,094 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:19,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:19,097 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:19,097 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,097 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,097 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,097 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:19,099 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:19,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5129c616[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,104 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:19,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:19,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:19,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:19,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,109 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:19,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:19,109 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,110 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,110 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,110 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,112 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:19,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,116 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,116 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,116 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,215 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/6ae04b59f208478daef548b223e29fd7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/6ae04b59f208478daef548b223e29fd7 2023-04-22 14:55:19,216 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,220 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/B of 0c924aecfddd8b7ecdab058989640d2e into 6ae04b59f208478daef548b223e29fd7(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:19,220 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:19,220 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/B, priority=12, startTime=1682175318743; duration=0sec 2023-04-22 14:55:19,221 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:19,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:19,223 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:19,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:19,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:19,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:19,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:19,226 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:19,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:19,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:19,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:19,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb6ea4b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:19,231 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,234 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:19,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,235 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,239 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:19,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:19,239 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,241 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:19,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,248 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,248 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,248 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,347 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/1983aeb8642241f6a32506fdf468510d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1983aeb8642241f6a32506fdf468510d 2023-04-22 14:55:19,348 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,353 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/C of 0c924aecfddd8b7ecdab058989640d2e into 1983aeb8642241f6a32506fdf468510d(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:19,353 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:19,353 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/C, priority=12, startTime=1682175318743; duration=0sec 2023-04-22 14:55:19,353 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:19,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:19,364 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:19,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:19,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:19,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:19,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:19,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,381 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:19,381 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:19,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:19,439 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224bc04c25acec46d995024b676a4dc402 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224bc04c25acec46d995024b676a4dc402 2023-04-22 14:55:19,449 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=91, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/9d833cbc416f4a2698f0d454d02316d3 2023-04-22 14:55:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:19,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:19,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175319638, max timestamp=1682175319672Num uniques 859; 2023-04-22 14:55:19,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:19,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175319638, max timestamp=1682175319672Num uniques 859; 2023-04-22 14:55:19,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:19,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175319638, max timestamp=1682175319672Num uniques 859; 2023-04-22 14:55:19,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175319763, max timestamp=1682175319797Num uniques 1018; 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175319763, max timestamp=1682175319797Num uniques 1018; 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175319763, max timestamp=1682175319797Num uniques 1018; 2023-04-22 14:55:19,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:55:19,892 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/8871dfd4dc5b44c68c5a394bffee3d3c 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175319869, max timestamp=1682175319896Num uniques 1077; 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175319869, max timestamp=1682175319896Num uniques 1077; 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175319869, max timestamp=1682175319896Num uniques 1077; 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-22 14:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-22 14:55:19,914 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/2f924a7d6e4049ba9618b231ab17c800 2023-04-22 14:55:19,932 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/9d833cbc416f4a2698f0d454d02316d3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3 2023-04-22 14:55:19,939 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3, entries=50, sequenceid=91, filesize=11.6 K 2023-04-22 14:55:19,940 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/8871dfd4dc5b44c68c5a394bffee3d3c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c 2023-04-22 14:55:19,945 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c, entries=50, sequenceid=91, filesize=7.0 K 2023-04-22 14:55:19,946 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/2f924a7d6e4049ba9618b231ab17c800 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800 2023-04-22 14:55:19,953 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800, entries=50, sequenceid=91, filesize=7.0 K 2023-04-22 14:55:19,954 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=73.80 KB/75570 for 0c924aecfddd8b7ecdab058989640d2e in 573ms, sequenceid=91, compaction requested=true 2023-04-22 14:55:19,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:19,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:19,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=80.51 KB heapSize=223.92 KB 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:19,959 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbc4955[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:19,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,975 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:19,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:19,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:19,975 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,975 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:19,977 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:19,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,983 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:19,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:19,983 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,984 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,984 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,984 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,985 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,987 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:19,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:19,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:19,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:19,990 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:19,990 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,090 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,093 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,093 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,094 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175380085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,096 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,097 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,097 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,098 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,098 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,098 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,098 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,100 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159d44a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,104 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,104 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,113 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,113 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,116 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,122 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,122 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:20,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175380197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,222 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,225 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,227 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,229 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,229 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,229 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,229 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5fbb01[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,235 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,235 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,235 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,237 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,237 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,237 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,237 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,239 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,240 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,241 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,241 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,241 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,246 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,246 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,347 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,350 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,350 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,350 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,350 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,352 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,354 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,354 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,354 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,354 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,361 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b7ee33[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,365 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175380369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,376 DEBUG [Thread-2210] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,378 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,378 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175380373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,385 DEBUG [Thread-2206] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175380374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175380376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,390 DEBUG [Thread-2208] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:20,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,390 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,390 DEBUG [Thread-2214] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:20,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,390 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,391 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,391 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:20,411 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224ed67763df574addaf515bd917fd1bb1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ed67763df574addaf515bd917fd1bb1 2023-04-22 14:55:20,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175380403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,417 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/263445a9845f466e8797ad7edcb9d3aa 2023-04-22 14:55:20,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/46920617b0b24740a92c098fca466059 2023-04-22 14:55:20,491 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,497 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,500 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb8fbd1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,510 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,510 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,517 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,517 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,517 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,517 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,517 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,517 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,617 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,620 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,622 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,627 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248c9968[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,630 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,636 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,636 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,641 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:20,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,643 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,644 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,644 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:20,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50324 deadline: 1682175380715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,744 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,747 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,748 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,749 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,750 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,750 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,751 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:20,751 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:20,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e8a94b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:20,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:20,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:20,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:20,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,762 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:20,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:20,762 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,763 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,765 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:20,772 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:20,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:20,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,773 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,773 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,847 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/78631e38af31453ebef3c2e83772913e 2023-04-22 14:55:20,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/263445a9845f466e8797ad7edcb9d3aa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa 2023-04-22 14:55:20,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa, entries=150, sequenceid=154, filesize=25.3 K 2023-04-22 14:55:20,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/46920617b0b24740a92c098fca466059 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059 2023-04-22 14:55:20,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059, entries=150, sequenceid=154, filesize=11.8 K 2023-04-22 14:55:20,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/78631e38af31453ebef3c2e83772913e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e 2023-04-22 14:55:20,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e, entries=150, sequenceid=154, filesize=11.8 K 2023-04-22 14:55:20,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~223.88 KB/229248, currentSize=100.63 KB/103050 for 0c924aecfddd8b7ecdab058989640d2e in 914ms, sequenceid=154, compaction requested=true 2023-04-22 14:55:20,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:20,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:20,872 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:20,872 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:20,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:20,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:20,873 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,875 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43146 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:20,875 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/B is initiating minor compaction (all files) 2023-04-22 14:55:20,875 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/B in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,875 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/6ae04b59f208478daef548b223e29fd7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=42.1 K 2023-04-22 14:55:20,875 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 89382 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:20,875 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/A is initiating minor compaction (all files) 2023-04-22 14:55:20,875 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/A in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,875 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/83ea88d3c2284282a504c32d200a7724, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=87.3 K 2023-04-22 14:55:20,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,877 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 83ea88d3c2284282a504c32d200a7724, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316222 2023-04-22 14:55:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:20,877 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6ae04b59f208478daef548b223e29fd7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316222 2023-04-22 14:55:20,878 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0f19c85d0c60455bbc8a88e833965f71, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175317091 2023-04-22 14:55:20,878 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3372c0f8435d450488c52a6a0aa223f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175317091 2023-04-22 14:55:20,878 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9d833cbc416f4a2698f0d454d02316d3, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682175319323 2023-04-22 14:55:20,878 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8871dfd4dc5b44c68c5a394bffee3d3c, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682175319323 2023-04-22 14:55:20,878 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 263445a9845f466e8797ad7edcb9d3aa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:20,878 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:20,879 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 46920617b0b24740a92c098fca466059, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,885 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:20,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:20,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:20,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:20,888 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:20,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:20,890 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#B#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:20,895 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#A#compaction#317 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:20,920 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/107aef5c8ef84efebf1a3f52122b6ecd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/107aef5c8ef84efebf1a3f52122b6ecd 2023-04-22 14:55:20,927 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/B of 0c924aecfddd8b7ecdab058989640d2e into 107aef5c8ef84efebf1a3f52122b6ecd(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:20,927 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:20,927 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/B, priority=12, startTime=1682175320872; duration=0sec 2023-04-22 14:55:20,928 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:20,928 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:20,930 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43146 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:20,931 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/C is initiating minor compaction (all files) 2023-04-22 14:55:20,931 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/C in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:20,931 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1983aeb8642241f6a32506fdf468510d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=42.1 K 2023-04-22 14:55:20,931 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1983aeb8642241f6a32506fdf468510d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682175316222 2023-04-22 14:55:20,932 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 75655502333c42ae9eb4244a96e2b4c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175317091 2023-04-22 14:55:20,933 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/bb3bbf2269814605b21e2cdb0a37e021 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/bb3bbf2269814605b21e2cdb0a37e021 2023-04-22 14:55:20,933 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2f924a7d6e4049ba9618b231ab17c800, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682175319323 2023-04-22 14:55:20,934 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 78631e38af31453ebef3c2e83772913e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:20,943 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/A of 0c924aecfddd8b7ecdab058989640d2e into bb3bbf2269814605b21e2cdb0a37e021(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:20,943 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:20,943 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/A, priority=12, startTime=1682175320872; duration=0sec 2023-04-22 14:55:20,944 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:20,944 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#C#compaction#320 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:21,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:21,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175381274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,317 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042276b77a570590480ea5b64321e2ce14a4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042276b77a570590480ea5b64321e2ce14a4 2023-04-22 14:55:21,322 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0 2023-04-22 14:55:21,335 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/66568d8af140446c98862cddc0dedd9e 2023-04-22 14:55:21,351 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ee48f6d24fa848f1ac729cb823735e65 2023-04-22 14:55:21,355 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/3878a34e74714979b11b162ddee808b3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3878a34e74714979b11b162ddee808b3 2023-04-22 14:55:21,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0 2023-04-22 14:55:21,372 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/C of 0c924aecfddd8b7ecdab058989640d2e into 3878a34e74714979b11b162ddee808b3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:21,372 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:21,372 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/C, priority=12, startTime=1682175320872; duration=0sec 2023-04-22 14:55:21,372 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:21,375 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0, entries=150, sequenceid=172, filesize=25.3 K 2023-04-22 14:55:21,376 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/66568d8af140446c98862cddc0dedd9e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e 2023-04-22 14:55:21,381 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e, entries=150, sequenceid=172, filesize=11.8 K 2023-04-22 14:55:21,383 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ee48f6d24fa848f1ac729cb823735e65 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65 2023-04-22 14:55:21,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175381382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,389 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65, entries=150, sequenceid=172, filesize=11.8 K 2023-04-22 14:55:21,390 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0c924aecfddd8b7ecdab058989640d2e in 502ms, sequenceid=172, compaction requested=false 2023-04-22 14:55:21,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:21,390 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:21,393 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:21,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8301f3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:21,396 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:21,396 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,397 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:21,397 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,397 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:21,399 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:21,402 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:21,402 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:21,403 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:21,403 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:21,403 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:21,407 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,407 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,407 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,409 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:21,412 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,419 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:21,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:21,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:21,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,419 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:21,420 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:21,520 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:21,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:21,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:21,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:21,524 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:21,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:21,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:21,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:21,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:21,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:21,528 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:21,528 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:21,528 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:21,529 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:21,535 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,536 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,536 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:21,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:21,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175381665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175381772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,955 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a14b7b06cdfc4d07afe1e9e008f7aec0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a14b7b06cdfc4d07afe1e9e008f7aec0 2023-04-22 14:55:21,968 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/7a8555e9d8b34ae38c995ec178230abb 2023-04-22 14:55:21,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175381977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:21,991 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/85a925301cfa4b0fbc596691bea2d40f 2023-04-22 14:55:22,018 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/d7d1c028b2064e9cb9c43b6941b00f16 2023-04-22 14:55:22,022 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/7a8555e9d8b34ae38c995ec178230abb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb 2023-04-22 14:55:22,030 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb, entries=150, sequenceid=189, filesize=25.3 K 2023-04-22 14:55:22,031 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/85a925301cfa4b0fbc596691bea2d40f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f 2023-04-22 14:55:22,038 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f, entries=150, sequenceid=189, filesize=11.8 K 2023-04-22 14:55:22,039 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/d7d1c028b2064e9cb9c43b6941b00f16 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16 2023-04-22 14:55:22,044 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16, entries=150, sequenceid=189, filesize=11.8 K 2023-04-22 14:55:22,044 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0c924aecfddd8b7ecdab058989640d2e in 516ms, sequenceid=189, compaction requested=true 2023-04-22 14:55:22,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:22,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:22,050 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c91901[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:22,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:22,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:22,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:22,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,054 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:22,057 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:22,057 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:22,057 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:22,057 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:22,057 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:22,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,058 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,059 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,063 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,063 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:22,064 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:22,064 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:22,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:22,164 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:22,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:22,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:22,170 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:22,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:22,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:22,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:22,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:22,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:22,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:22,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:22,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:22,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:22,174 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:22,174 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:22,189 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042214898f9b585e4fc8a78b49de58cee67b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042214898f9b585e4fc8a78b49de58cee67b 2023-04-22 14:55:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:22,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:22,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:22,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175382352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175382460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,607 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/1dbf2e747f744143bb15aba1a70a8a0c 2023-04-22 14:55:22,649 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/49338a752ac34ca5b41867793c445ded 2023-04-22 14:55:22,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:22,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175382665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:22,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175382968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,070 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/39ed6134f7ad4c6190fde3b694dbc09c 2023-04-22 14:55:23,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/1dbf2e747f744143bb15aba1a70a8a0c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c 2023-04-22 14:55:23,081 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c, entries=150, sequenceid=207, filesize=25.3 K 2023-04-22 14:55:23,081 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/49338a752ac34ca5b41867793c445ded as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded 2023-04-22 14:55:23,086 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded, entries=150, sequenceid=207, filesize=11.8 K 2023-04-22 14:55:23,087 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/39ed6134f7ad4c6190fde3b694dbc09c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c 2023-04-22 14:55:23,091 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c, entries=150, sequenceid=207, filesize=11.8 K 2023-04-22 14:55:23,096 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0c924aecfddd8b7ecdab058989640d2e in 922ms, sequenceid=207, compaction requested=true 2023-04-22 14:55:23,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:23,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:23,098 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:23,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,098 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce8f69[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:23,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:23,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,102 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:23,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:23,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:23,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:23,103 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:23,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:23,106 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:23,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:23,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:23,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:23,106 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:23,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:23,109 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,109 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,112 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:23,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:23,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:23,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:23,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:23,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:23,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,125 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:23,125 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:23,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,225 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:23,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:23,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:23,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:23,229 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:23,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:23,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:23,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:23,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:23,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:23,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:23,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:23,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:23,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:23,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:23,242 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:23,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:23,256 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422bf2c9db5a5c2462ebbd93f3b8aee9f01 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bf2c9db5a5c2462ebbd93f3b8aee9f01 2023-04-22 14:55:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:23,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:23,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:23,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175383575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,661 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/24e09c0b2e8a48ee891028efcea7965e 2023-04-22 14:55:23,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:23,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175383682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:23,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:23,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175383889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,072 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/5bcd2490c94c44a8a7154afc53f32e1a 2023-04-22 14:55:24,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175384196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50378 deadline: 1682175384389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,395 DEBUG [Thread-2210] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:24,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50360 deadline: 1682175384397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,404 DEBUG [Thread-2206] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:24,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50312 deadline: 1682175384403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,412 DEBUG [Thread-2208] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:24,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50374 deadline: 1682175384423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,432 DEBUG [Thread-2214] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:24,481 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/c51ac3e7e4e34ffbb1cefabb12d22f50 2023-04-22 14:55:24,486 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/24e09c0b2e8a48ee891028efcea7965e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e 2023-04-22 14:55:24,489 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e, entries=150, sequenceid=221, filesize=25.3 K 2023-04-22 14:55:24,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/5bcd2490c94c44a8a7154afc53f32e1a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a 2023-04-22 14:55:24,498 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a, entries=150, sequenceid=221, filesize=11.8 K 2023-04-22 14:55:24,499 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/c51ac3e7e4e34ffbb1cefabb12d22f50 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50 2023-04-22 14:55:24,503 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50, entries=150, sequenceid=221, filesize=11.8 K 2023-04-22 14:55:24,505 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0c924aecfddd8b7ecdab058989640d2e in 1263ms, sequenceid=221, compaction requested=true 2023-04-22 14:55:24,505 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:24,505 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:24,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:24,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,508 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:24,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:24,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eedc0a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:24,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:24,510 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,513 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:24,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,515 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:24,516 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:24,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:24,525 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:24,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:24,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:24,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:24,526 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:24,528 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,528 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:24,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:24,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:24,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,531 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:24,535 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:24,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:24,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:24,535 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:24,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:24,535 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:24,637 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:24,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:24,643 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:24,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:24,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:24,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:24,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:24,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:24,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:24,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:24,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:24,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:24,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:24,656 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:24,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:24,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175384802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:24,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175384921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,090 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042254ccde4a735640c4b79533a206b5ad82 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042254ccde4a735640c4b79533a206b5ad82 2023-04-22 14:55:25,099 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/f40afed362ce4df8bc1aa4c245bd58d6 2023-04-22 14:55:25,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:25,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175385133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:25,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175385442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,511 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/2a338daf73f64f0fa7a7c5757ae0262a 2023-04-22 14:55:25,538 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/f0f7b87ca285428b822ca8032536b198 2023-04-22 14:55:25,546 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/f40afed362ce4df8bc1aa4c245bd58d6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6 2023-04-22 14:55:25,551 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6, entries=150, sequenceid=239, filesize=25.3 K 2023-04-22 14:55:25,553 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/2a338daf73f64f0fa7a7c5757ae0262a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a 2023-04-22 14:55:25,557 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a, entries=150, sequenceid=239, filesize=11.8 K 2023-04-22 14:55:25,558 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/f0f7b87ca285428b822ca8032536b198 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198 2023-04-22 14:55:25,562 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198, entries=150, sequenceid=239, filesize=11.8 K 2023-04-22 14:55:25,563 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0c924aecfddd8b7ecdab058989640d2e in 907ms, sequenceid=239, compaction requested=true 2023-04-22 14:55:25,563 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:25,563 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:25,565 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:25,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:25,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7610d9fd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:25,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:25,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,568 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:25,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:25,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:25,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:25,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:25,570 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:25,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:25,573 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:25,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:25,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:25,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:25,573 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:25,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:25,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:25,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:25,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:25,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:25,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:25,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:25,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:25,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:25,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:25,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:25,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:25,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:25,580 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:25,580 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:25,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:25,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:25,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:25,680 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:25,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:25,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:25,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:25,685 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:25,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:25,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:25,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:25,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:25,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,686 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:25,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:25,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:25,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:25,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:25,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:25,692 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:25,692 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:25,692 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:25,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:25,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:25,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:25,695 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:25,696 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:55:25,696 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:25,697 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:25,697 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:25,697 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:25,697 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:25,697 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:25,720 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223be4a40aef11443d9dbb3e6719f2c0d5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223be4a40aef11443d9dbb3e6719f2c0d5 2023-04-22 14:55:25,738 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/6bf9ee1510f4480993d39f5db2151822 2023-04-22 14:55:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:25,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:26,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:26,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175386030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175386139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,163 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/f6b6c921faec443083faa41790f4e94d 2023-04-22 14:55:26,176 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/15011e8bfbed451fabca11d8a20a2c35 2023-04-22 14:55:26,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/6bf9ee1510f4480993d39f5db2151822 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822 2023-04-22 14:55:26,184 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822, entries=150, sequenceid=253, filesize=25.3 K 2023-04-22 14:55:26,185 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/f6b6c921faec443083faa41790f4e94d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d 2023-04-22 14:55:26,188 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d, entries=150, sequenceid=253, filesize=11.8 K 2023-04-22 14:55:26,189 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/15011e8bfbed451fabca11d8a20a2c35 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35 2023-04-22 14:55:26,191 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35, entries=150, sequenceid=253, filesize=11.8 K 2023-04-22 14:55:26,192 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0c924aecfddd8b7ecdab058989640d2e in 496ms, sequenceid=253, compaction requested=true 2023-04-22 14:55:26,192 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:26,192 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:26,195 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781fd72f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:26,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:26,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,200 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:26,201 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,201 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:26,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:26,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:26,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:26,203 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:26,207 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:26,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:26,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:26,207 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:26,207 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:26,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:26,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:26,215 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:26,216 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:26,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:26,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:26,220 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:26,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:26,220 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:26,320 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:26,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:26,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:26,323 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:26,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:26,329 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:26,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:26,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:26,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:26,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:26,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:26,331 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:26,331 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:26,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:26,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175386420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:26,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175386529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,744 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220de37d77ffab4b2e8063ff15b3a8dcc0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220de37d77ffab4b2e8063ff15b3a8dcc0 2023-04-22 14:55:26,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175386739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:26,749 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/f78e45290ac8499689e59ea06149af4b 2023-04-22 14:55:27,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:27,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175387048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,187 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/f7fcaec8fc35470e8094da9c35adef02 2023-04-22 14:55:27,197 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/fae862181b2b4a349951eed4543ef41b 2023-04-22 14:55:27,200 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/f78e45290ac8499689e59ea06149af4b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b 2023-04-22 14:55:27,204 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b, entries=150, sequenceid=271, filesize=25.4 K 2023-04-22 14:55:27,205 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/f7fcaec8fc35470e8094da9c35adef02 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02 2023-04-22 14:55:27,208 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02, entries=150, sequenceid=271, filesize=11.9 K 2023-04-22 14:55:27,209 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/fae862181b2b4a349951eed4543ef41b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b 2023-04-22 14:55:27,212 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b, entries=150, sequenceid=271, filesize=11.9 K 2023-04-22 14:55:27,213 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0c924aecfddd8b7ecdab058989640d2e in 882ms, sequenceid=271, compaction requested=true 2023-04-22 14:55:27,213 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:27,213 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:27,217 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,217 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:27,218 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,218 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad8ba82[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:27,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:27,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,220 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:27,220 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:27,220 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,220 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:27,220 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:27,222 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:27,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:27,224 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:27,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:27,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:27,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:27,224 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:27,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:27,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:27,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,230 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:27,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:27,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:27,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:27,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:27,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:27,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:27,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:27,234 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:27,234 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:27,334 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:27,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:27,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:27,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:27,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:27,337 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:27,337 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:27,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:27,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:27,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:27,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:27,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:27,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:27,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:27,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:27,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:27,340 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:27,341 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:27,341 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:27,342 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:27,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:27,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:27,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175387708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:27,760 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042230dfeba012d14f14868c718332df75e5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042230dfeba012d14f14868c718332df75e5 2023-04-22 14:55:27,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:27,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175387816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:28,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175388026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,190 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/5590db02e06b44479f35551570c13c35 2023-04-22 14:55:28,206 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/77137a2d9b9d4565987f00318a624dce 2023-04-22 14:55:28,227 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/553293bf4b514d39838d6a37897948b2 2023-04-22 14:55:28,231 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/5590db02e06b44479f35551570c13c35 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35 2023-04-22 14:55:28,236 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35, entries=150, sequenceid=285, filesize=25.4 K 2023-04-22 14:55:28,237 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/77137a2d9b9d4565987f00318a624dce as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce 2023-04-22 14:55:28,244 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce, entries=150, sequenceid=285, filesize=11.9 K 2023-04-22 14:55:28,245 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/553293bf4b514d39838d6a37897948b2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2 2023-04-22 14:55:28,250 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2, entries=150, sequenceid=285, filesize=11.9 K 2023-04-22 14:55:28,251 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0c924aecfddd8b7ecdab058989640d2e in 910ms, sequenceid=285, compaction requested=true 2023-04-22 14:55:28,251 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:28,251 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,254 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7824295f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,261 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,262 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,263 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,263 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,263 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,266 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,266 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,266 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,267 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,268 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,274 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,274 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,274 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:28,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:28,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:28,374 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:28,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:28,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:28,378 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:28,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:28,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:28,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:28,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:28,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,382 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:28,383 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:28,383 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,383 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,384 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88bc897[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,387 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,387 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,389 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,391 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,392 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,394 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,394 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,395 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,398 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,398 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,398 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,398 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,400 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:28,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175388476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,498 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:28,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:28,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:28,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:28,507 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:28,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:28,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:28,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,509 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:28,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:28,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:28,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:28,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,512 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,521 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2bc006[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,523 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,523 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,524 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,525 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,525 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,525 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,528 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,528 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,532 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,535 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,535 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,535 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:28,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175388582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,636 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:28,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:28,639 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:28,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:28,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:28,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,641 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,641 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,641 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:28,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,643 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:28,644 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,644 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:28,644 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:28,644 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416867eb[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,649 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,649 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,649 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,651 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,651 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,651 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,654 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,654 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,654 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,654 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,654 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,656 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,659 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,659 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,659 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,659 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,660 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,752 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422bebe5581bf254cfa957121e654e25d33 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bebe5581bf254cfa957121e654e25d33 2023-04-22 14:55:28,758 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/c922a32745e648a9884eeed1f71c7197 2023-04-22 14:55:28,759 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:28,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:28,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:28,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:28,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:28,763 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:28,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:28,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:28,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,766 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:28,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:28,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:28,771 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,771 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:28,771 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:28,771 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,771 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185fd048[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,776 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,776 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,776 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,780 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,780 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,780 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,783 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,786 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,787 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,787 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,787 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,787 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175388791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,887 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:28,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:28,891 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:28,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:28,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:28,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,893 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,894 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,894 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:28,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:28,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:28,896 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,896 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:28,896 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:28,896 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:28,897 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:28,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec66ca4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:28,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:28,899 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:28,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:28,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,908 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:28,910 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:28,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:28,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:28,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:28,911 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:28,913 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,913 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,913 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:28,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:28,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:28,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,917 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:28,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:28,917 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:28,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,017 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:29,021 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:29,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:29,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:29,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,021 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:29,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:29,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:29,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:29,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:29,027 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebdbe8a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:29,027 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,028 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:29,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:29,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:29,034 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:29,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:29,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:29,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:29,034 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:29,036 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,036 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,036 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:29,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,040 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:29,040 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175389104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,140 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:29,144 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:29,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:29,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:29,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,145 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:29,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:29,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,147 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,147 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:29,147 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:29,147 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:29,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:29,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce8d589[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:29,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:29,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:29,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,155 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:29,159 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:29,159 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:29,159 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:29,159 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:29,159 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:29,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,160 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,162 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,165 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:29,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,166 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:29,166 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:29,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/adcdc28f27b44d0b938769ffd2d9f41f 2023-04-22 14:55:29,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/757b2bdd4e84482bbdf0f11ab0866cdc 2023-04-22 14:55:29,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/c922a32745e648a9884eeed1f71c7197 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197 2023-04-22 14:55:29,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197, entries=200, sequenceid=304, filesize=32.3 K 2023-04-22 14:55:29,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/adcdc28f27b44d0b938769ffd2d9f41f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f 2023-04-22 14:55:29,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 14:55:29,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/757b2bdd4e84482bbdf0f11ab0866cdc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc 2023-04-22 14:55:29,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc, entries=150, sequenceid=304, filesize=11.9 K 2023-04-22 14:55:29,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0c924aecfddd8b7ecdab058989640d2e in 863ms, sequenceid=304, compaction requested=true 2023-04-22 14:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:29,201 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-22 14:55:29,201 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-22 14:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:29,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 266724 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-22 14:55:29,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/A is initiating minor compaction (all files) 2023-04-22 14:55:29,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/A in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/bb3bbf2269814605b21e2cdb0a37e021, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=260.5 K 2023-04-22 14:55:29,204 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121084 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-22 14:55:29,205 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/B is initiating minor compaction (all files) 2023-04-22 14:55:29,205 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/B in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,205 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/107aef5c8ef84efebf1a3f52122b6ecd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=118.2 K 2023-04-22 14:55:29,205 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 107aef5c8ef84efebf1a3f52122b6ecd, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:29,205 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting bb3bbf2269814605b21e2cdb0a37e021, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:29,206 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 66568d8af140446c98862cddc0dedd9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1682175320060 2023-04-22 14:55:29,206 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3a2ed6fb1c4c49e6b7003e0c4a86e3d0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1682175320060 2023-04-22 14:55:29,206 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 85a925301cfa4b0fbc596691bea2d40f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682175321267 2023-04-22 14:55:29,206 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 49338a752ac34ca5b41867793c445ded, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1682175321643 2023-04-22 14:55:29,206 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7a8555e9d8b34ae38c995ec178230abb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682175321267 2023-04-22 14:55:29,207 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5bcd2490c94c44a8a7154afc53f32e1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682175322313 2023-04-22 14:55:29,207 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 1dbf2e747f744143bb15aba1a70a8a0c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1682175321643 2023-04-22 14:55:29,207 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2a338daf73f64f0fa7a7c5757ae0262a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682175323523 2023-04-22 14:55:29,207 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 24e09c0b2e8a48ee891028efcea7965e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682175322313 2023-04-22 14:55:29,207 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f6b6c921faec443083faa41790f4e94d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682175324723 2023-04-22 14:55:29,208 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f40afed362ce4df8bc1aa4c245bd58d6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682175323523 2023-04-22 14:55:29,208 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f7fcaec8fc35470e8094da9c35adef02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682175325987 2023-04-22 14:55:29,208 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6bf9ee1510f4480993d39f5db2151822, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682175324723 2023-04-22 14:55:29,208 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 77137a2d9b9d4565987f00318a624dce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175326387 2023-04-22 14:55:29,216 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting adcdc28f27b44d0b938769ffd2d9f41f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175327680 2023-04-22 14:55:29,216 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f78e45290ac8499689e59ea06149af4b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682175325987 2023-04-22 14:55:29,217 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5590db02e06b44479f35551570c13c35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682175326387 2023-04-22 14:55:29,217 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c922a32745e648a9884eeed1f71c7197, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175327662 2023-04-22 14:55:29,236 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#B#compaction#348 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:29,244 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#A#compaction#347 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:29,266 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:29,270 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:29,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:29,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:29,271 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,271 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:29,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:29,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:29,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,276 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:29,276 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:29,276 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,292 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304220010d39101d249a0bec0d8eb543659b0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220010d39101d249a0bec0d8eb543659b0 2023-04-22 14:55:29,306 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/e4f8df29ead64488afe15696d8d0175f 2023-04-22 14:55:29,354 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/827bd5200352411f92e7391310fa5904 2023-04-22 14:55:29,390 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ca3327f7256a45d5bc8808e323d2aa8b 2023-04-22 14:55:29,395 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/e4f8df29ead64488afe15696d8d0175f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f 2023-04-22 14:55:29,400 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f, entries=150, sequenceid=317, filesize=25.4 K 2023-04-22 14:55:29,401 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/827bd5200352411f92e7391310fa5904 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904 2023-04-22 14:55:29,410 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904, entries=150, sequenceid=317, filesize=11.9 K 2023-04-22 14:55:29,411 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/ca3327f7256a45d5bc8808e323d2aa8b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b 2023-04-22 14:55:29,418 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b, entries=150, sequenceid=317, filesize=11.9 K 2023-04-22 14:55:29,419 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 0c924aecfddd8b7ecdab058989640d2e in 143ms, sequenceid=317, compaction requested=true 2023-04-22 14:55:29,419 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,419 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:29,421 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,421 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad39bcc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:29,423 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:29,424 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,426 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:29,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:29,427 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:29,427 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,427 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,427 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:29,430 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:29,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:29,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:29,431 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:29,431 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:29,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,436 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,438 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,442 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,442 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,442 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,442 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,442 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:29,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,442 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:29,443 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,443 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:29,443 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:29,443 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,443 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,443 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,543 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:29,552 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:29,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:29,552 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:29,553 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,553 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,555 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,555 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,556 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,556 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:29,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:29,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:29,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d6b7e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:29,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:29,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,578 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:29,578 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,578 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:29,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:29,581 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,581 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:29,584 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:29,584 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:29,584 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:29,584 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:29,584 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:29,586 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,586 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,586 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,587 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,591 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,591 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,591 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:29,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,592 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:29,592 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:29,591 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:29,593 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,652 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/63e698c6305a4c8dbdf5d1080aa42851 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/63e698c6305a4c8dbdf5d1080aa42851 2023-04-22 14:55:29,658 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/B of 0c924aecfddd8b7ecdab058989640d2e into 63e698c6305a4c8dbdf5d1080aa42851(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:29,658 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,658 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/B, priority=6, startTime=1682175329201; duration=0sec 2023-04-22 14:55:29,658 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:29,658 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120960 starting at candidate #1 after considering 44 permutations with 44 in ratio 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/C is initiating minor compaction 2023-04-22 14:55:29,663 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/C in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,663 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=118.1 K 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ee48f6d24fa848f1ac729cb823735e65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d7d1c028b2064e9cb9c43b6941b00f16, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 39ed6134f7ad4c6190fde3b694dbc09c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c51ac3e7e4e34ffbb1cefabb12d22f50, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f0f7b87ca285428b822ca8032536b198, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 15011e8bfbed451fabca11d8a20a2c35, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253 2023-04-22 14:55:29,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting fae862181b2b4a349951eed4543ef41b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271 2023-04-22 14:55:29,664 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 553293bf4b514d39838d6a37897948b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285 2023-04-22 14:55:29,664 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 757b2bdd4e84482bbdf0f11ab0866cdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304 2023-04-22 14:55:29,664 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ca3327f7256a45d5bc8808e323d2aa8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317 2023-04-22 14:55:29,687 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#C#compaction#352 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:29,692 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,709 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/faf242c39450431baf69b29dfa0b91fb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/faf242c39450431baf69b29dfa0b91fb 2023-04-22 14:55:29,715 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/A of 0c924aecfddd8b7ecdab058989640d2e into faf242c39450431baf69b29dfa0b91fb(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:29,715 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,715 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/A, priority=6, startTime=1682175329201; duration=0sec 2023-04-22 14:55:29,715 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:29,716 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:29,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:29,716 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:29,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,718 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:29,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:29,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:29,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:29,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:29,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:29,724 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:29,725 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-22 14:55:29,725 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:29,725 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,725 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:29,726 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,726 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:29,726 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:29,730 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/da1263e7c821423e8804a723e5866396 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/da1263e7c821423e8804a723e5866396 2023-04-22 14:55:29,735 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 0c924aecfddd8b7ecdab058989640d2e/C of 0c924aecfddd8b7ecdab058989640d2e into da1263e7c821423e8804a723e5866396(size=12.2 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:29,736 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:29,736 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/C, priority=5, startTime=1682175329201; duration=0sec 2023-04-22 14:55:29,736 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:29,761 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304221e73d5ace2574e67a357f0b398d41eb2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221e73d5ace2574e67a357f0b398d41eb2 2023-04-22 14:55:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:29,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:30,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175330059, max timestamp=1682175330099Num uniques 859; 2023-04-22 14:55:30,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175330059, max timestamp=1682175330099Num uniques 859; 2023-04-22 14:55:30,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:30,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175330059, max timestamp=1682175330099Num uniques 859; 2023-04-22 14:55:30,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:30,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:30,199 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/d3edafd0d6cc4b4e8fe9f36de1496337 2023-04-22 14:55:30,227 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/150d506df7dd4773920e88ec3fa92fde 2023-04-22 14:55:30,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175330166, max timestamp=1682175330237Num uniques 1018; 2023-04-22 14:55:30,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175330166, max timestamp=1682175330237Num uniques 1018; 2023-04-22 14:55:30,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175330166, max timestamp=1682175330237Num uniques 1018; 2023-04-22 14:55:30,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:55:30,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:55:30,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:55:30,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175330307, max timestamp=1682175330356Num uniques 1027; 2023-04-22 14:55:30,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:55:30,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175330307, max timestamp=1682175330356Num uniques 1027; 2023-04-22 14:55:30,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:55:30,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175330307, max timestamp=1682175330356Num uniques 1027; 2023-04-22 14:55:30,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:55:30,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175330415, max timestamp=1682175330467Num uniques 1036; 2023-04-22 14:55:30,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:55:30,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175330415, max timestamp=1682175330467Num uniques 1036; 2023-04-22 14:55:30,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:55:30,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175330415, max timestamp=1682175330467Num uniques 1036; 2023-04-22 14:55:30,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175330558, max timestamp=1682175330578Num uniques 1045; 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175330558, max timestamp=1682175330578Num uniques 1045; 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175330558, max timestamp=1682175330578Num uniques 1045; 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:55:30,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:55:30,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175330625, max timestamp=1682175330664Num uniques 1054; 2023-04-22 14:55:30,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175330625, max timestamp=1682175330664Num uniques 1054; 2023-04-22 14:55:30,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175330625, max timestamp=1682175330664Num uniques 1054; 2023-04-22 14:55:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:30,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:30,686 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/9699e8945c914ec5bc216a6fd9a3a2fd 2023-04-22 14:55:30,691 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/d3edafd0d6cc4b4e8fe9f36de1496337 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337 2023-04-22 14:55:30,698 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337, entries=50, sequenceid=324, filesize=11.7 K 2023-04-22 14:55:30,699 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/150d506df7dd4773920e88ec3fa92fde as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde 2023-04-22 14:55:30,703 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde, entries=50, sequenceid=324, filesize=7.1 K 2023-04-22 14:55:30,704 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/9699e8945c914ec5bc216a6fd9a3a2fd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd 2023-04-22 14:55:30,707 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd, entries=50, sequenceid=324, filesize=7.1 K 2023-04-22 14:55:30,708 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=60.38 KB/61830 for 0c924aecfddd8b7ecdab058989640d2e in 984ms, sequenceid=324, compaction requested=true 2023-04-22 14:55:30,708 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:30,708 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:30,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,710 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74b7dcfb[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:30,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:30,719 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,719 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,720 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:30,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,720 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=67.09 KB heapSize=181.55 KB 2023-04-22 14:55:30,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:30,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:30,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:30,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:30,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:30,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:30,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:30,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:30,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:30,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:30,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:30,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:30,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:30,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,751 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:30,751 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:30,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,759 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ded954579d9b46058129790cbee4f305 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ded954579d9b46058129790cbee4f305 2023-04-22 14:55:30,851 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:30,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175390846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:30,854 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:30,855 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,855 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,856 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,856 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:30,858 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:30,858 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:30,858 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:30,858 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:30,858 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,858 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,858 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,858 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:30,858 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:30,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,859 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:30,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea90b79[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,864 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:30,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:30,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:30,867 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:30,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:30,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:30,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:30,868 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:30,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,870 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,871 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,875 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,875 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:30,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:30,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,876 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:30,876 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:30,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175390953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,976 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:30,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:30,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:30,979 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:30,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:30,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:30,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,981 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,982 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:30,982 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:30,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:30,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:30,984 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:30,984 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:30,984 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:30,984 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:30,984 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:30,990 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c2f89b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:30,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:30,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,997 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:31,001 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:31,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:31,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:31,001 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:31,001 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:31,002 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,003 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,009 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:31,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:31,009 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:31,009 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,009 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,109 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:31,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:31,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:31,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:31,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:31,113 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:31,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:31,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:31,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,113 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,115 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:31,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:31,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:31,117 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,117 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:31,117 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:31,117 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:31,117 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:31,118 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:31,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24783c66[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:31,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:31,122 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,124 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:31,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:31,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:31,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,124 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:31,127 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:31,135 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:31,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:31,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:31,135 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:31,135 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:31,143 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,145 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:31,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:31,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:31,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,149 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:31,149 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:31,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175391160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,166 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=437, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/528275ac31bd4d189c0593f9e3e0fa1e 2023-04-22 14:55:31,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/8d7d143cd8404806b9efaef9b31721f1 2023-04-22 14:55:31,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/f499e1cdd60d4fbdb662a7e225dffbcb 2023-04-22 14:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/528275ac31bd4d189c0593f9e3e0fa1e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e 2023-04-22 14:55:31,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e, entries=200, sequenceid=437, filesize=32.3 K 2023-04-22 14:55:31,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/8d7d143cd8404806b9efaef9b31721f1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1 2023-04-22 14:55:31,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1, entries=150, sequenceid=437, filesize=11.9 K 2023-04-22 14:55:31,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/f499e1cdd60d4fbdb662a7e225dffbcb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb 2023-04-22 14:55:31,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb, entries=150, sequenceid=437, filesize=11.9 K 2023-04-22 14:55:31,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.50 KB/185856, currentSize=114.05 KB/116790 for 0c924aecfddd8b7ecdab058989640d2e in 510ms, sequenceid=437, compaction requested=true 2023-04-22 14:55:31,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:31,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:31,233 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:31,233 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:31,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:31,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:31,234 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97748 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:31,234 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44472 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:31,234 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/A is initiating minor compaction (all files) 2023-04-22 14:55:31,234 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/B is initiating minor compaction (all files) 2023-04-22 14:55:31,234 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/A in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,234 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/B in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,234 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/faf242c39450431baf69b29dfa0b91fb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=95.5 K 2023-04-22 14:55:31,234 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/63e698c6305a4c8dbdf5d1080aa42851, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=43.4 K 2023-04-22 14:55:31,235 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting faf242c39450431baf69b29dfa0b91fb, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175327680 2023-04-22 14:55:31,235 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 63e698c6305a4c8dbdf5d1080aa42851, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682175327680 2023-04-22 14:55:31,235 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e4f8df29ead64488afe15696d8d0175f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682175328408 2023-04-22 14:55:31,235 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 827bd5200352411f92e7391310fa5904, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682175328408 2023-04-22 14:55:31,236 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d3edafd0d6cc4b4e8fe9f36de1496337, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1682175329710 2023-04-22 14:55:31,236 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 150d506df7dd4773920e88ec3fa92fde, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1682175329710 2023-04-22 14:55:31,236 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 528275ac31bd4d189c0593f9e3e0fa1e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682175330679 2023-04-22 14:55:31,236 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8d7d143cd8404806b9efaef9b31721f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682175330679 2023-04-22 14:55:31,249 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:31,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:31,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:31,254 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:31,254 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:31,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:31,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:31,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:31,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:31,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:31,270 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#A#compaction#359 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:31,270 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:31,270 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:31,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,270 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:31,270 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:31,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:31,278 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/4151ab50694e4f68a2dfafc5c31b90b1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/4151ab50694e4f68a2dfafc5c31b90b1 2023-04-22 14:55:31,282 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/B of 0c924aecfddd8b7ecdab058989640d2e into 4151ab50694e4f68a2dfafc5c31b90b1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:31,282 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:31,282 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/B, priority=12, startTime=1682175331233; duration=0sec 2023-04-22 14:55:31,282 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:31,282 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:31,283 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:31,283 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 0c924aecfddd8b7ecdab058989640d2e/C is initiating minor compaction (all files) 2023-04-22 14:55:31,283 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c924aecfddd8b7ecdab058989640d2e/C in TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:31,283 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3878a34e74714979b11b162ddee808b3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/da1263e7c821423e8804a723e5866396, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp, totalSize=43.3 K 2023-04-22 14:55:31,283 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3878a34e74714979b11b162ddee808b3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682175319920 2023-04-22 14:55:31,284 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting da1263e7c821423e8804a723e5866396, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682175328408 2023-04-22 14:55:31,284 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9699e8945c914ec5bc216a6fd9a3a2fd, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1682175329710 2023-04-22 14:55:31,285 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f499e1cdd60d4fbdb662a7e225dffbcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682175330679 2023-04-22 14:55:31,306 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 0c924aecfddd8b7ecdab058989640d2e#C#compaction#362 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:31,327 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/1fa64edf35be4104ae8cc3ec87d0ca2d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1fa64edf35be4104ae8cc3ec87d0ca2d 2023-04-22 14:55:31,330 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/C of 0c924aecfddd8b7ecdab058989640d2e into 1fa64edf35be4104ae8cc3ec87d0ca2d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:31,330 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:31,330 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/C, priority=12, startTime=1682175331233; duration=0sec 2023-04-22 14:55:31,331 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:31,468 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:31,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:31,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175391499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:31,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175391603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:31,713 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ce831068cac843ffbc40a2e1e502ed9c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce831068cac843ffbc40a2e1e502ed9c 2023-04-22 14:55:31,719 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=457, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/cf513f39042f4452a3059f7fc7d0cf70 2023-04-22 14:55:31,729 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/6f2112adecb342c69a5e0ed7f957c480 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6f2112adecb342c69a5e0ed7f957c480 2023-04-22 14:55:31,732 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/e31a17224e9e44fa8ef3ae6da00d3562 2023-04-22 14:55:31,734 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0c924aecfddd8b7ecdab058989640d2e/A of 0c924aecfddd8b7ecdab058989640d2e into 6f2112adecb342c69a5e0ed7f957c480(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:31,734 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:31,734 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e., storeName=0c924aecfddd8b7ecdab058989640d2e/A, priority=12, startTime=1682175331233; duration=0sec 2023-04-22 14:55:31,734 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:31,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:31,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175391808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175392111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,144 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/3858b5b2cb8f4c7889c8a83d78ec68ed 2023-04-22 14:55:32,148 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/cf513f39042f4452a3059f7fc7d0cf70 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/cf513f39042f4452a3059f7fc7d0cf70 2023-04-22 14:55:32,151 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/cf513f39042f4452a3059f7fc7d0cf70, entries=150, sequenceid=457, filesize=25.4 K 2023-04-22 14:55:32,152 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/e31a17224e9e44fa8ef3ae6da00d3562 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/e31a17224e9e44fa8ef3ae6da00d3562 2023-04-22 14:55:32,155 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/e31a17224e9e44fa8ef3ae6da00d3562, entries=150, sequenceid=457, filesize=11.9 K 2023-04-22 14:55:32,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/3858b5b2cb8f4c7889c8a83d78ec68ed as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3858b5b2cb8f4c7889c8a83d78ec68ed 2023-04-22 14:55:32,159 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3858b5b2cb8f4c7889c8a83d78ec68ed, entries=150, sequenceid=457, filesize=11.9 K 2023-04-22 14:55:32,160 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0c924aecfddd8b7ecdab058989640d2e in 890ms, sequenceid=457, compaction requested=false 2023-04-22 14:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:32,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:32,162 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee27c16[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:32,165 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:32,165 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:32,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,167 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:32,170 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:32,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:32,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:32,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:32,170 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:32,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,174 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:32,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:32,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:32,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:32,178 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:32,178 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:32,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,278 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:32,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:32,282 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:32,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:32,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:32,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:32,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:32,286 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,286 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:32,287 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,287 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:32,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:32,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:32,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:32,288 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:32,288 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. started... 2023-04-22 14:55:32,289 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:32,289 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:32,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. as already flushing 2023-04-22 14:55:32,702 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304225571d03d575a41f2b7b45a8b291977c1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225571d03d575a41f2b7b45a8b291977c1 2023-04-22 14:55:32,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175392704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175392811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:32,843 DEBUG [Thread-2219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:51853 2023-04-22 14:55:32,843 DEBUG [Thread-2219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:32,843 DEBUG [Thread-2217] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:51853 2023-04-22 14:55:32,843 DEBUG [Thread-2217] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:32,844 DEBUG [Thread-2223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:51853 2023-04-22 14:55:32,844 DEBUG [Thread-2223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:32,848 DEBUG [Thread-2221] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:51853 2023-04-22 14:55:32,848 DEBUG [Thread-2221] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:32,849 DEBUG [Thread-2225] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:51853 2023-04-22 14:55:32,849 DEBUG [Thread-2225] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:33,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:33,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175393015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,107 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=472, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/85faf616dd2b477984507e2e40ee10ef 2023-04-22 14:55:33,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:33,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175393317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,516 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/9a4c4455813e45d2a7ec7787f4d2ec22 2023-04-22 14:55:33,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50324 deadline: 1682175393821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,924 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/96bff0ed3fc94a429d9b5bd8ba3687eb 2023-04-22 14:55:33,928 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/85faf616dd2b477984507e2e40ee10ef as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/85faf616dd2b477984507e2e40ee10ef 2023-04-22 14:55:33,930 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/85faf616dd2b477984507e2e40ee10ef, entries=150, sequenceid=472, filesize=25.4 K 2023-04-22 14:55:33,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/9a4c4455813e45d2a7ec7787f4d2ec22 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/9a4c4455813e45d2a7ec7787f4d2ec22 2023-04-22 14:55:33,934 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/9a4c4455813e45d2a7ec7787f4d2ec22, entries=150, sequenceid=472, filesize=11.9 K 2023-04-22 14:55:33,934 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/96bff0ed3fc94a429d9b5bd8ba3687eb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/96bff0ed3fc94a429d9b5bd8ba3687eb 2023-04-22 14:55:33,937 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/96bff0ed3fc94a429d9b5bd8ba3687eb, entries=150, sequenceid=472, filesize=11.9 K 2023-04-22 14:55:33,938 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0c924aecfddd8b7ecdab058989640d2e in 1650ms, sequenceid=472, compaction requested=true 2023-04-22 14:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:33,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:33,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8764070[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:33,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:33,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:33,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:33,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:33,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:33,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:33,947 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:33,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:33,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:33,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:33,947 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:33,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:33,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:33,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:33,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:33,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:33,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:33,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:33,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:33,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:33,952 DEBUG [Thread-2216] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,952 DEBUG [Thread-2216] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:33,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:33,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:34,053 DEBUG [Thread-2216] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:34,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:34,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:34,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-22 14:55:34,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:34,441 DEBUG [Thread-2214] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:51853 2023-04-22 14:55:34,441 DEBUG [Thread-2214] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:34,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:34,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:34,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:34,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:34,450 DEBUG [Thread-2210] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:51853 2023-04-22 14:55:34,450 DEBUG [Thread-2210] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,455 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422040641cf0abd4f588e841a7b99118fe8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422040641cf0abd4f588e841a7b99118fe8 2023-04-22 14:55:34,474 DEBUG [Thread-2206] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:51853 2023-04-22 14:55:34,474 DEBUG [Thread-2206] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,484 DEBUG [Thread-2208] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:51853 2023-04-22 14:55:34,484 DEBUG [Thread-2208] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,832 DEBUG [Thread-2212] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:51853 2023-04-22 14:55:34,833 DEBUG [Thread-2212] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 362 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1747 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5241 rows 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1744 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5229 rows 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1749 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5247 rows 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1767 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5301 rows 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1748 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 5244 rows 2023-04-22 14:55:34,833 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:55:34,834 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:51853 2023-04-22 14:55:34,834 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:34,836 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:55:34,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:55:34,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:34,843 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175334843"}]},"ts":"1682175334843"} 2023-04-22 14:55:34,846 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:55:34,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:34,848 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:55:34,849 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, UNASSIGN}] 2023-04-22 14:55:34,851 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, UNASSIGN 2023-04-22 14:55:34,851 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:34,851 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175334851"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175334851"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175334851"}]},"ts":"1682175334851"} 2023-04-22 14:55:34,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:34,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/6ed0f5572cc94a4ead71230892fa5188 2023-04-22 14:55:34,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:35,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:35,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0c924aecfddd8b7ecdab058989640d2e, disabling compactions & flushes 2023-04-22 14:55:35,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:35,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:35,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/1bbee0d48a5e49c1bc776e402a090ff6 2023-04-22 14:55:35,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:35,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/c128422e560f4d10a1229bb9bee626a9 2023-04-22 14:55:35,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/6ed0f5572cc94a4ead71230892fa5188 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6ed0f5572cc94a4ead71230892fa5188 2023-04-22 14:55:35,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6ed0f5572cc94a4ead71230892fa5188, entries=150, sequenceid=493, filesize=25.4 K 2023-04-22 14:55:35,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/1bbee0d48a5e49c1bc776e402a090ff6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/1bbee0d48a5e49c1bc776e402a090ff6 2023-04-22 14:55:35,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/1bbee0d48a5e49c1bc776e402a090ff6, entries=150, sequenceid=493, filesize=11.9 K 2023-04-22 14:55:35,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/c128422e560f4d10a1229bb9bee626a9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c128422e560f4d10a1229bb9bee626a9 2023-04-22 14:55:35,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c128422e560f4d10a1229bb9bee626a9, entries=150, sequenceid=493, filesize=11.9 K 2023-04-22 14:55:35,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for 0c924aecfddd8b7ecdab058989640d2e in 1257ms, sequenceid=493, compaction requested=true 2023-04-22 14:55:35,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:35,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:35,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:35,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:35,698 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. because compaction request was cancelled 2023-04-22 14:55:35,698 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. because compaction request was cancelled 2023-04-22 14:55:35,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. after waiting 0 ms 2023-04-22 14:55:35,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:35,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:35,698 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. because compaction request was cancelled 2023-04-22 14:55:35,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0c924aecfddd8b7ecdab058989640d2e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 14:55:35,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=A 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=B 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0c924aecfddd8b7ecdab058989640d2e, store=C 2023-04-22 14:55:35,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:35,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d20055a1b835427e8e30ffd8d58e8ced to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d20055a1b835427e8e30ffd8d58e8ced 2023-04-22 14:55:35,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:36,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/05813afa990546cb9110e1f5ea2461c2 2023-04-22 14:55:36,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/da0b4f17d10b4fab80b9415ef6149507 2023-04-22 14:55:36,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/dbd09781106f478b9300edda91fceed6 2023-04-22 14:55:36,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/A/05813afa990546cb9110e1f5ea2461c2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/05813afa990546cb9110e1f5ea2461c2 2023-04-22 14:55:36,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/05813afa990546cb9110e1f5ea2461c2, entries=100, sequenceid=500, filesize=18.6 K 2023-04-22 14:55:36,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/B/da0b4f17d10b4fab80b9415ef6149507 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/da0b4f17d10b4fab80b9415ef6149507 2023-04-22 14:55:36,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/da0b4f17d10b4fab80b9415ef6149507, entries=100, sequenceid=500, filesize=9.5 K 2023-04-22 14:55:36,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/.tmp/C/dbd09781106f478b9300edda91fceed6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/dbd09781106f478b9300edda91fceed6 2023-04-22 14:55:36,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:36,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/dbd09781106f478b9300edda91fceed6, entries=100, sequenceid=500, filesize=9.5 K 2023-04-22 14:55:36,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0c924aecfddd8b7ecdab058989640d2e in 1254ms, sequenceid=500, compaction requested=true 2023-04-22 14:55:36,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/83ea88d3c2284282a504c32d200a7724, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/bb3bbf2269814605b21e2cdb0a37e021, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/faf242c39450431baf69b29dfa0b91fb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e] to archive 2023-04-22 14:55:36,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:36,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5dff9f1815b84cee8c82469e92c242c7 2023-04-22 14:55:36,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/86b484f4f4d44286a5df2d8339c91238 2023-04-22 14:55:36,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/8af4f97b513942d2bea94b578f54b131 2023-04-22 14:55:36,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3943591c34a1470d83383d2c0af16703 2023-04-22 14:55:36,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/83ea88d3c2284282a504c32d200a7724 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/83ea88d3c2284282a504c32d200a7724 2023-04-22 14:55:36,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/0f19c85d0c60455bbc8a88e833965f71 2023-04-22 14:55:36,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/9d833cbc416f4a2698f0d454d02316d3 2023-04-22 14:55:36,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/bb3bbf2269814605b21e2cdb0a37e021 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/bb3bbf2269814605b21e2cdb0a37e021 2023-04-22 14:55:36,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/263445a9845f466e8797ad7edcb9d3aa 2023-04-22 14:55:36,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/3a2ed6fb1c4c49e6b7003e0c4a86e3d0 2023-04-22 14:55:36,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/7a8555e9d8b34ae38c995ec178230abb 2023-04-22 14:55:36,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/1dbf2e747f744143bb15aba1a70a8a0c 2023-04-22 14:55:36,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/24e09c0b2e8a48ee891028efcea7965e 2023-04-22 14:55:36,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f40afed362ce4df8bc1aa4c245bd58d6 2023-04-22 14:55:36,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6bf9ee1510f4480993d39f5db2151822 2023-04-22 14:55:36,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/f78e45290ac8499689e59ea06149af4b 2023-04-22 14:55:36,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/5590db02e06b44479f35551570c13c35 2023-04-22 14:55:36,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/c922a32745e648a9884eeed1f71c7197 2023-04-22 14:55:36,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/faf242c39450431baf69b29dfa0b91fb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/faf242c39450431baf69b29dfa0b91fb 2023-04-22 14:55:36,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/e4f8df29ead64488afe15696d8d0175f 2023-04-22 14:55:36,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/d3edafd0d6cc4b4e8fe9f36de1496337 2023-04-22 14:55:36,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/528275ac31bd4d189c0593f9e3e0fa1e 2023-04-22 14:55:37,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/6ae04b59f208478daef548b223e29fd7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/107aef5c8ef84efebf1a3f52122b6ecd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/63e698c6305a4c8dbdf5d1080aa42851, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1] to archive 2023-04-22 14:55:37,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ddf9dc7eab53417ea88681d15af4e298 2023-04-22 14:55:37,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/a88771ae416347028843cdeacc52f27a 2023-04-22 14:55:37,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/ecd8da92a0a545e6a2c05bc11128026c 2023-04-22 14:55:37,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/6ae04b59f208478daef548b223e29fd7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/6ae04b59f208478daef548b223e29fd7 2023-04-22 14:55:37,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/08484b3ce1874e2699be6c032e616a84 2023-04-22 14:55:37,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/3372c0f8435d450488c52a6a0aa223f5 2023-04-22 14:55:37,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8871dfd4dc5b44c68c5a394bffee3d3c 2023-04-22 14:55:37,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/107aef5c8ef84efebf1a3f52122b6ecd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/107aef5c8ef84efebf1a3f52122b6ecd 2023-04-22 14:55:37,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/46920617b0b24740a92c098fca466059 2023-04-22 14:55:37,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/66568d8af140446c98862cddc0dedd9e 2023-04-22 14:55:37,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/85a925301cfa4b0fbc596691bea2d40f 2023-04-22 14:55:37,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/49338a752ac34ca5b41867793c445ded 2023-04-22 14:55:37,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/5bcd2490c94c44a8a7154afc53f32e1a 2023-04-22 14:55:37,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/2a338daf73f64f0fa7a7c5757ae0262a 2023-04-22 14:55:37,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f6b6c921faec443083faa41790f4e94d 2023-04-22 14:55:37,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/f7fcaec8fc35470e8094da9c35adef02 2023-04-22 14:55:37,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/77137a2d9b9d4565987f00318a624dce 2023-04-22 14:55:37,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/63e698c6305a4c8dbdf5d1080aa42851 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/63e698c6305a4c8dbdf5d1080aa42851 2023-04-22 14:55:37,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/adcdc28f27b44d0b938769ffd2d9f41f 2023-04-22 14:55:37,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/827bd5200352411f92e7391310fa5904 2023-04-22 14:55:37,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/150d506df7dd4773920e88ec3fa92fde 2023-04-22 14:55:37,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/8d7d143cd8404806b9efaef9b31721f1 2023-04-22 14:55:37,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1983aeb8642241f6a32506fdf468510d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3878a34e74714979b11b162ddee808b3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/da1263e7c821423e8804a723e5866396, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb] to archive 2023-04-22 14:55:37,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/17bd5f36b610440993f9e7caa1297b12 2023-04-22 14:55:37,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ab0857e7b321482696a220c50711604a 2023-04-22 14:55:37,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/582c7624a52b4150ae47dc909155644b 2023-04-22 14:55:37,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1983aeb8642241f6a32506fdf468510d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1983aeb8642241f6a32506fdf468510d 2023-04-22 14:55:37,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9946f61b6c5c4d71853ca2a1b97c5ac8 2023-04-22 14:55:37,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/75655502333c42ae9eb4244a96e2b4c9 2023-04-22 14:55:37,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/2f924a7d6e4049ba9618b231ab17c800 2023-04-22 14:55:37,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3878a34e74714979b11b162ddee808b3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3878a34e74714979b11b162ddee808b3 2023-04-22 14:55:37,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/78631e38af31453ebef3c2e83772913e 2023-04-22 14:55:37,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ee48f6d24fa848f1ac729cb823735e65 2023-04-22 14:55:37,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/d7d1c028b2064e9cb9c43b6941b00f16 2023-04-22 14:55:37,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/39ed6134f7ad4c6190fde3b694dbc09c 2023-04-22 14:55:37,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c51ac3e7e4e34ffbb1cefabb12d22f50 2023-04-22 14:55:37,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f0f7b87ca285428b822ca8032536b198 2023-04-22 14:55:37,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/15011e8bfbed451fabca11d8a20a2c35 2023-04-22 14:55:37,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/fae862181b2b4a349951eed4543ef41b 2023-04-22 14:55:37,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/553293bf4b514d39838d6a37897948b2 2023-04-22 14:55:37,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/757b2bdd4e84482bbdf0f11ab0866cdc 2023-04-22 14:55:37,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/da1263e7c821423e8804a723e5866396 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/da1263e7c821423e8804a723e5866396 2023-04-22 14:55:37,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/ca3327f7256a45d5bc8808e323d2aa8b 2023-04-22 14:55:37,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/9699e8945c914ec5bc216a6fd9a3a2fd 2023-04-22 14:55:37,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/f499e1cdd60d4fbdb662a7e225dffbcb 2023-04-22 14:55:37,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits/503.seqid, newMaxSeqId=503, maxSeqId=4 2023-04-22 14:55:37,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e. 2023-04-22 14:55:37,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0c924aecfddd8b7ecdab058989640d2e: 2023-04-22 14:55:37,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:37,052 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=0c924aecfddd8b7ecdab058989640d2e, regionState=CLOSED 2023-04-22 14:55:37,052 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175337052"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175337052"}]},"ts":"1682175337052"} 2023-04-22 14:55:37,054 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-22 14:55:37,054 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 0c924aecfddd8b7ecdab058989640d2e, server=jenkins-hbase4.apache.org,32935,1682175218140 in 2.2010 sec 2023-04-22 14:55:37,055 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-22 14:55:37,055 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0c924aecfddd8b7ecdab058989640d2e, UNASSIGN in 2.2050 sec 2023-04-22 14:55:37,056 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175337056"}]},"ts":"1682175337056"} 2023-04-22 14:55:37,057 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:55:37,060 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:55:37,062 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2250 sec 2023-04-22 14:55:38,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-22 14:55:38,952 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-22 14:55:38,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:55:38,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:38,955 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:38,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-22 14:55:38,955 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:38,959 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:38,961 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits] 2023-04-22 14:55:38,963 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/05813afa990546cb9110e1f5ea2461c2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/05813afa990546cb9110e1f5ea2461c2 2023-04-22 14:55:38,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6ed0f5572cc94a4ead71230892fa5188 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6ed0f5572cc94a4ead71230892fa5188 2023-04-22 14:55:38,965 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6f2112adecb342c69a5e0ed7f957c480 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/6f2112adecb342c69a5e0ed7f957c480 2023-04-22 14:55:38,966 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/85faf616dd2b477984507e2e40ee10ef to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/85faf616dd2b477984507e2e40ee10ef 2023-04-22 14:55:38,967 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/cf513f39042f4452a3059f7fc7d0cf70 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/A/cf513f39042f4452a3059f7fc7d0cf70 2023-04-22 14:55:38,969 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/1bbee0d48a5e49c1bc776e402a090ff6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/1bbee0d48a5e49c1bc776e402a090ff6 2023-04-22 14:55:38,970 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/4151ab50694e4f68a2dfafc5c31b90b1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/4151ab50694e4f68a2dfafc5c31b90b1 2023-04-22 14:55:38,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/9a4c4455813e45d2a7ec7787f4d2ec22 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/9a4c4455813e45d2a7ec7787f4d2ec22 2023-04-22 14:55:38,972 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/da0b4f17d10b4fab80b9415ef6149507 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/da0b4f17d10b4fab80b9415ef6149507 2023-04-22 14:55:38,972 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/e31a17224e9e44fa8ef3ae6da00d3562 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/B/e31a17224e9e44fa8ef3ae6da00d3562 2023-04-22 14:55:38,974 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1fa64edf35be4104ae8cc3ec87d0ca2d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/1fa64edf35be4104ae8cc3ec87d0ca2d 2023-04-22 14:55:38,975 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3858b5b2cb8f4c7889c8a83d78ec68ed to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/3858b5b2cb8f4c7889c8a83d78ec68ed 2023-04-22 14:55:38,976 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/96bff0ed3fc94a429d9b5bd8ba3687eb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/96bff0ed3fc94a429d9b5bd8ba3687eb 2023-04-22 14:55:38,977 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c128422e560f4d10a1229bb9bee626a9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/c128422e560f4d10a1229bb9bee626a9 2023-04-22 14:55:38,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/dbd09781106f478b9300edda91fceed6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/C/dbd09781106f478b9300edda91fceed6 2023-04-22 14:55:38,979 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits/503.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e/recovered.edits/503.seqid 2023-04-22 14:55:38,980 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/0c924aecfddd8b7ecdab058989640d2e 2023-04-22 14:55:38,980 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:55:38,980 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:55:38,981 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 14:55:38,983 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220010d39101d249a0bec0d8eb543659b0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220010d39101d249a0bec0d8eb543659b0 2023-04-22 14:55:38,984 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422040641cf0abd4f588e841a7b99118fe8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422040641cf0abd4f588e841a7b99118fe8 2023-04-22 14:55:38,984 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220de37d77ffab4b2e8063ff15b3a8dcc0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304220de37d77ffab4b2e8063ff15b3a8dcc0 2023-04-22 14:55:38,985 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042214898f9b585e4fc8a78b49de58cee67b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042214898f9b585e4fc8a78b49de58cee67b 2023-04-22 14:55:38,986 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221e73d5ace2574e67a357f0b398d41eb2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304221e73d5ace2574e67a357f0b398d41eb2 2023-04-22 14:55:38,987 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042230dfeba012d14f14868c718332df75e5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042230dfeba012d14f14868c718332df75e5 2023-04-22 14:55:38,988 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223be4a40aef11443d9dbb3e6719f2c0d5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223be4a40aef11443d9dbb3e6719f2c0d5 2023-04-22 14:55:38,989 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042242cc23a61d5248109d93ee121c9d1479 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042242cc23a61d5248109d93ee121c9d1479 2023-04-22 14:55:38,989 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224bc04c25acec46d995024b676a4dc402 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224bc04c25acec46d995024b676a4dc402 2023-04-22 14:55:38,990 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ed67763df574addaf515bd917fd1bb1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224ed67763df574addaf515bd917fd1bb1 2023-04-22 14:55:38,991 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042254ccde4a735640c4b79533a206b5ad82 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042254ccde4a735640c4b79533a206b5ad82 2023-04-22 14:55:38,992 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225571d03d575a41f2b7b45a8b291977c1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225571d03d575a41f2b7b45a8b291977c1 2023-04-22 14:55:38,992 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042266d85439d2d44c1f927c0c5f2f5f19e5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042266d85439d2d44c1f927c0c5f2f5f19e5 2023-04-22 14:55:38,993 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042276b77a570590480ea5b64321e2ce14a4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042276b77a570590480ea5b64321e2ce14a4 2023-04-22 14:55:38,994 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277129ff3161e4267946b1fd2fa2c324b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042277129ff3161e4267946b1fd2fa2c324b 2023-04-22 14:55:38,994 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a14b7b06cdfc4d07afe1e9e008f7aec0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a14b7b06cdfc4d07afe1e9e008f7aec0 2023-04-22 14:55:38,995 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b508e11eb6784c1594ee82a6a48d185d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b508e11eb6784c1594ee82a6a48d185d 2023-04-22 14:55:38,996 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bebe5581bf254cfa957121e654e25d33 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bebe5581bf254cfa957121e654e25d33 2023-04-22 14:55:38,997 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bf2c9db5a5c2462ebbd93f3b8aee9f01 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bf2c9db5a5c2462ebbd93f3b8aee9f01 2023-04-22 14:55:38,997 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c91036c3452d459d90fac685db571190 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c91036c3452d459d90fac685db571190 2023-04-22 14:55:38,998 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce831068cac843ffbc40a2e1e502ed9c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ce831068cac843ffbc40a2e1e502ed9c 2023-04-22 14:55:38,999 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d20055a1b835427e8e30ffd8d58e8ced to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d20055a1b835427e8e30ffd8d58e8ced 2023-04-22 14:55:39,000 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ded954579d9b46058129790cbee4f305 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ded954579d9b46058129790cbee4f305 2023-04-22 14:55:39,000 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:55:39,002 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:39,003 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:55:39,004 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:55:39,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:39,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:55:39,005 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175339005"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:39,006 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:55:39,006 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0c924aecfddd8b7ecdab058989640d2e, NAME => 'TestAcidGuarantees,,1682175309761.0c924aecfddd8b7ecdab058989640d2e.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:55:39,007 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:55:39,007 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175339007"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:39,008 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:55:39,009 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:39,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-04-22 14:55:39,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-22 14:55:39,057 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-22 14:55:39,069 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 225) - Thread LEAK? -, OpenFileDescriptor=550 (was 548) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 349), ProcessCount=170 (was 172), AvailableMemoryMB=12290 (was 12439) 2023-04-22 14:55:39,081 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=550, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=170, AvailableMemoryMB=12289 2023-04-22 14:55:39,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:55:39,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:55:39,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:55:39,086 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:39,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-22 14:55:39,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:39,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:39,088 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:39,088 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 empty. 2023-04-22 14:55:39,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:39,089 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:55:39,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:39,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:39,484 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:55:39,496 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:39,497 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 95b9d53f66a350cbea7732aecb64c699, NAME => 'TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:55:39,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 95b9d53f66a350cbea7732aecb64c699, disabling compactions & flushes 2023-04-22 14:55:39,910 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. after waiting 0 ms 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:39,910 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:39,910 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:39,913 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:39,914 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175339914"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175339914"}]},"ts":"1682175339914"} 2023-04-22 14:55:39,915 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:39,915 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:39,916 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175339916"}]},"ts":"1682175339916"} 2023-04-22 14:55:39,916 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:55:39,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, ASSIGN}] 2023-04-22 14:55:39,921 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, ASSIGN 2023-04-22 14:55:39,922 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:55:40,073 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=95b9d53f66a350cbea7732aecb64c699, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:40,073 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175340073"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175340073"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175340073"}]},"ts":"1682175340073"} 2023-04-22 14:55:40,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:55:40,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:40,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:40,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 95b9d53f66a350cbea7732aecb64c699, NAME => 'TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:40,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:40,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,232 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,233 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A 2023-04-22 14:55:40,233 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A 2023-04-22 14:55:40,233 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:40,233 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 95b9d53f66a350cbea7732aecb64c699 columnFamilyName A 2023-04-22 14:55:40,234 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(310): Store=95b9d53f66a350cbea7732aecb64c699/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:40,234 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,235 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B 2023-04-22 14:55:40,235 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B 2023-04-22 14:55:40,235 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:40,235 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 95b9d53f66a350cbea7732aecb64c699 columnFamilyName B 2023-04-22 14:55:40,236 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(310): Store=95b9d53f66a350cbea7732aecb64c699/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:40,236 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,237 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C 2023-04-22 14:55:40,237 DEBUG [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C 2023-04-22 14:55:40,237 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:55:40,237 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 95b9d53f66a350cbea7732aecb64c699 columnFamilyName C 2023-04-22 14:55:40,238 INFO [StoreOpener-95b9d53f66a350cbea7732aecb64c699-1] regionserver.HStore(310): Store=95b9d53f66a350cbea7732aecb64c699/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:40,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:40,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:55:40,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:40,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:40,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 95b9d53f66a350cbea7732aecb64c699; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71872491, jitterRate=0.07098357379436493}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:55:40,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:40,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., pid=49, masterSystemTime=1682175340226 2023-04-22 14:55:40,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:40,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:40,246 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=95b9d53f66a350cbea7732aecb64c699, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:40,246 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175340246"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175340246"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175340246"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175340246"}]},"ts":"1682175340246"} 2023-04-22 14:55:40,251 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-22 14:55:40,251 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 in 172 msec 2023-04-22 14:55:40,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-22 14:55:40,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, ASSIGN in 331 msec 2023-04-22 14:55:40,253 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:40,253 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175340253"}]},"ts":"1682175340253"} 2023-04-22 14:55:40,254 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:55:40,256 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:40,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-22 14:55:41,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-22 14:55:41,193 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-22 14:55:41,193 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,199 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,201 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,203 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,204 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:55:41,207 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:55:41,208 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,213 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,213 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,218 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,218 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,228 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,228 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,235 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,235 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,243 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,243 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,251 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,252 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,260 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,260 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,271 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,271 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,279 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,279 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x653bb8fc to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:55:41,291 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b8f652e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:55:41,319 DEBUG [hconnection-0x7fa518b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,319 DEBUG [hconnection-0x2e8e7be7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,319 DEBUG [hconnection-0x6016122c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,320 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,323 DEBUG [hconnection-0x3a6d9a73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,323 DEBUG [hconnection-0x64c44334-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,324 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,324 DEBUG [hconnection-0x66f9e929-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,328 DEBUG [hconnection-0xc26ad98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,328 DEBUG [hconnection-0x6680783a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,329 DEBUG [hconnection-0x4d755e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,329 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,334 DEBUG [hconnection-0x5d0cb38b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:55:41,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:41,341 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,341 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,342 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,351 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,351 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,352 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,352 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46866, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:55:41,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:41,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:41,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:41,363 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:41,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:41,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:41,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:41,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:55:41,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:41,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:41,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:41,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:41,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:41,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:41,366 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:41,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:41,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21014, totalHeapSize=47976, min timestamp=1682175341371, max timestamp=1682175341379Num uniques 859; 2023-04-22 14:55:41,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=21014, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-22 14:55:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21014, totalHeapSize=47976, min timestamp=1682175341371, max timestamp=1682175341379Num uniques 859; 2023-04-22 14:55:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682175341371, max timestamp=1682175341379Num uniques 859; 2023-04-22 14:55:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=21014, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-22 14:55:41,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-22 14:55:41,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175401390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175401390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175401390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:41,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:41,395 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:41,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:41,402 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:41,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175401402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175401402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,406 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:41,406 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:41,406 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:41,406 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,420 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:41,420 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:41,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:41,423 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572f67c5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:41,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:41,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,428 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:41,429 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,429 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:41,430 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/f9f3b5a1eab54447a35453589a30f972 2023-04-22 14:55:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:41,433 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:41,433 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:41,433 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:41,433 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:41,433 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:41,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:41,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:41,439 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,439 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,439 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,442 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:41,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:41,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:41,453 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:41,453 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:41,453 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:41,453 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:41,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:41,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/48e22656d7524ea382ff988704f8974c 2023-04-22 14:55:41,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175401491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175401491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175401491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3f16341f9b654aed9a136600b96a6fb2 2023-04-22 14:55:41,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/f9f3b5a1eab54447a35453589a30f972 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972 2023-04-22 14:55:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175401506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175401507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972, entries=150, sequenceid=13, filesize=11.6 K 2023-04-22 14:55:41,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/48e22656d7524ea382ff988704f8974c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c 2023-04-22 14:55:41,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c, entries=150, sequenceid=13, filesize=11.6 K 2023-04-22 14:55:41,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3f16341f9b654aed9a136600b96a6fb2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2 2023-04-22 14:55:41,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2, entries=150, sequenceid=13, filesize=11.6 K 2023-04-22 14:55:41,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 95b9d53f66a350cbea7732aecb64c699 in 171ms, sequenceid=13, compaction requested=false 2023-04-22 14:55:41,534 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-22 14:55:41,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:41,554 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:41,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:41,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:41,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:41,564 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:41,564 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:41,564 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:41,564 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:41,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,566 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:41,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:41,566 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:41,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:41,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:41,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:41,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:41,572 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:41,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:41,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:41,572 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=127.47 KB heapSize=336.94 KB 2023-04-22 14:55:41,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:41,572 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:41,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:41,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175401701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175401702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175401702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175401710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175401710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175401803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175401803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:41,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175401804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,005 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/e6081b82374d4b319c7c8952198b4197 2023-04-22 14:55:42,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175402013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175402013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175402013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175402022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175402024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175402320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175402322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175402322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,467 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/8de943aa4a064bb7bc9fdc4236726adf 2023-04-22 14:55:42,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175402527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175402530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,634 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:55:42,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175402830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175402830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:42,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175402831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,890 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/56a0812e039c415fa7178e0e82376585 2023-04-22 14:55:42,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/e6081b82374d4b319c7c8952198b4197 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197 2023-04-22 14:55:42,898 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197, entries=150, sequenceid=43, filesize=11.6 K 2023-04-22 14:55:42,899 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/8de943aa4a064bb7bc9fdc4236726adf as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf 2023-04-22 14:55:42,903 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf, entries=150, sequenceid=43, filesize=11.6 K 2023-04-22 14:55:42,903 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/56a0812e039c415fa7178e0e82376585 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585 2023-04-22 14:55:42,908 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585, entries=150, sequenceid=43, filesize=11.6 K 2023-04-22 14:55:42,910 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~336.89 KB/344976, currentSize=67.09 KB/68700 for 95b9d53f66a350cbea7732aecb64c699 in 1338ms, sequenceid=43, compaction requested=false 2023-04-22 14:55:42,910 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:42,910 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:42,912 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,912 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:42,913 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,913 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:42,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:42,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d1fa4a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:42,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:42,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:42,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:42,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:42,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:42,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:42,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:42,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:42,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:42,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:42,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:42,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:42,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:42,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:42,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:42,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:42,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:42,945 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:42,945 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:43,045 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:43,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:43,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:43,049 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:43,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:43,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:43,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:43,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:43,052 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:43,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:43,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:43,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:43,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:43,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:43,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:43,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:43,060 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:43,060 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:43,060 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:43,060 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:43,060 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:55:43,060 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:43,479 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/6c221bce741a4503a57fa63137dd015a 2023-04-22 14:55:43,492 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/c66e3a7da06d401f8526fef0cbcc1226 2023-04-22 14:55:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:43,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:43,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175403586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175403587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175403689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175403698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175403837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175403844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175403845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175403893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:43,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175403902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,910 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/a424bd286a3f4fd9b513d9876371fc75 2023-04-22 14:55:43,914 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/6c221bce741a4503a57fa63137dd015a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a 2023-04-22 14:55:43,917 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a, entries=150, sequenceid=56, filesize=11.6 K 2023-04-22 14:55:43,918 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/c66e3a7da06d401f8526fef0cbcc1226 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226 2023-04-22 14:55:43,922 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226, entries=150, sequenceid=56, filesize=11.6 K 2023-04-22 14:55:43,922 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/a424bd286a3f4fd9b513d9876371fc75 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75 2023-04-22 14:55:43,926 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75, entries=150, sequenceid=56, filesize=11.6 K 2023-04-22 14:55:43,927 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 95b9d53f66a350cbea7732aecb64c699 in 866ms, sequenceid=56, compaction requested=true 2023-04-22 14:55:43,927 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:43,927 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:43,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:43,928 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fae32ca[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:43,930 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:43,930 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,931 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:43,932 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,932 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:43,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:43,937 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:43,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:43,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:43,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:43,937 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:43,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:43,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:43,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:43,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:43,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:43,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:43,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:43,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:43,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:43,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:43,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:43,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:43,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:43,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:43,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:43,953 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:43,953 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:43,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,053 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,081 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,094 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,095 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:44,095 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:44,096 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,116 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/9d03938c223d4418b0ec15b332638bba 2023-04-22 14:55:44,136 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/2d1ba3039698444d9b90e3c4e20eadd3 2023-04-22 14:55:44,160 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/cf97e6a7e6154fa28f5a746382b163ab 2023-04-22 14:55:44,173 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/9d03938c223d4418b0ec15b332638bba as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba 2023-04-22 14:55:44,176 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba, entries=150, sequenceid=75, filesize=11.6 K 2023-04-22 14:55:44,177 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-22 14:55:44,178 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/2d1ba3039698444d9b90e3c4e20eadd3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3 2023-04-22 14:55:44,189 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3, entries=150, sequenceid=75, filesize=11.6 K 2023-04-22 14:55:44,189 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/cf97e6a7e6154fa28f5a746382b163ab as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab 2023-04-22 14:55:44,193 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab, entries=150, sequenceid=75, filesize=11.6 K 2023-04-22 14:55:44,194 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 99ms, sequenceid=75, compaction requested=true 2023-04-22 14:55:44,194 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:44,194 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,198 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75618e8f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,203 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,203 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,205 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,205 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,205 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,207 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,211 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,211 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,211 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,211 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,211 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:44,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,226 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,226 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/570f33d025d14ff99496f8ef1ba3a0d5 2023-04-22 14:55:44,272 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/ecf0989249c04e92b6096d7f864d060d 2023-04-22 14:55:44,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175344283, max timestamp=1682175344310Num uniques 859; 2023-04-22 14:55:44,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:44,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175344283, max timestamp=1682175344310Num uniques 859; 2023-04-22 14:55:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175344283, max timestamp=1682175344310Num uniques 859; 2023-04-22 14:55:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:44,327 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,339 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,344 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,346 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:44,356 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,356 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,359 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595ccea9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,362 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,362 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,363 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,365 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,366 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,366 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,369 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,369 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,377 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,380 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,380 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175404399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175404405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,480 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,484 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,486 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:44,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e27c62[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,502 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,503 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,503 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,504 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,504 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,504 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,505 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,512 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,512 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175404517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175404521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,612 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,617 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:44,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,623 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae42a56[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,626 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,626 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,636 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,636 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,643 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,643 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/1203a5804a204feeb1ab3f14367c7846 2023-04-22 14:55:44,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/570f33d025d14ff99496f8ef1ba3a0d5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5 2023-04-22 14:55:44,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5, entries=200, sequenceid=85, filesize=13.9 K 2023-04-22 14:55:44,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/ecf0989249c04e92b6096d7f864d060d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d 2023-04-22 14:55:44,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d, entries=150, sequenceid=85, filesize=11.6 K 2023-04-22 14:55:44,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/1203a5804a204feeb1ab3f14367c7846 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846 2023-04-22 14:55:44,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846, entries=150, sequenceid=85, filesize=11.6 K 2023-04-22 14:55:44,718 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 493ms, sequenceid=85, compaction requested=true 2023-04-22 14:55:44,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:44,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:44,718 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:55:44,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:44,718 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:55:44,721 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:44,725 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:55:44,725 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:44,725 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,725 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61770 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:55:44,725 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=58.0 K 2023-04-22 14:55:44,725 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:44,726 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,726 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=60.3 K 2023-04-22 14:55:44,726 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 48e22656d7524ea382ff988704f8974c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682175341345 2023-04-22 14:55:44,726 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f9f3b5a1eab54447a35453589a30f972, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682175341345 2023-04-22 14:55:44,726 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8de943aa4a064bb7bc9fdc4236726adf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1682175341379 2023-04-22 14:55:44,727 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e6081b82374d4b319c7c8952198b4197, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1682175341379 2023-04-22 14:55:44,727 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c66e3a7da06d401f8526fef0cbcc1226, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1682175341697 2023-04-22 14:55:44,727 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6c221bce741a4503a57fa63137dd015a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1682175341697 2023-04-22 14:55:44,727 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2d1ba3039698444d9b90e3c4e20eadd3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682175343577 2023-04-22 14:55:44,728 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9d03938c223d4418b0ec15b332638bba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682175343577 2023-04-22 14:55:44,728 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ecf0989249c04e92b6096d7f864d060d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:44,728 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 570f33d025d14ff99496f8ef1ba3a0d5, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:44,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:44,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:44,744 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,750 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#390 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:44,754 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#391 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:44,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/5e5c924e58f24da89ceb13d5ac76f909 2023-04-22 14:55:44,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,762 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,768 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175404776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,785 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,785 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,785 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,785 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,785 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:44,785 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,785 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,785 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,785 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,788 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,788 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175404779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,793 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,793 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2308c273[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,795 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,795 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,796 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,798 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/cc93b097ca6a4d62a81d1d929d965ee3 2023-04-22 14:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,801 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/be89f40483b842b4a9d90dfeddacc662 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/be89f40483b842b4a9d90dfeddacc662 2023-04-22 14:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,803 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,804 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,812 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into be89f40483b842b4a9d90dfeddacc662(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:44,812 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:44,812 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=11, startTime=1682175344718; duration=0sec 2023-04-22 14:55:44,812 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:44,812 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:55:44,813 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:55:44,813 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:44,813 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,814 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=58.0 K 2023-04-22 14:55:44,814 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3f16341f9b654aed9a136600b96a6fb2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682175341345 2023-04-22 14:55:44,814 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 56a0812e039c415fa7178e0e82376585, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1682175341379 2023-04-22 14:55:44,815 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a424bd286a3f4fd9b513d9876371fc75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1682175341697 2023-04-22 14:55:44,815 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting cf97e6a7e6154fa28f5a746382b163ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682175343577 2023-04-22 14:55:44,815 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1203a5804a204feeb1ab3f14367c7846, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,819 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,819 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,819 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,820 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:44,830 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#394 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:44,842 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/d07292a8351c4207805fc67f428ccf4d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d07292a8351c4207805fc67f428ccf4d 2023-04-22 14:55:44,847 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into d07292a8351c4207805fc67f428ccf4d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:44,847 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:44,847 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=11, startTime=1682175344721; duration=0sec 2023-04-22 14:55:44,847 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:44,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175404880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175404893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,920 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:44,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:44,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:44,923 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:44,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:44,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:44,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,924 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,924 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:44,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:44,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:44,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:44,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:44,931 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:44,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:44,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1e793d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:44,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:44,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,936 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:44,936 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:44,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:44,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:44,945 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:44,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:44,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:44,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:44,946 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:44,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:44,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,952 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:44,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:44,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:44,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:44,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:44,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:44,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:44,954 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:44,954 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:44,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:45,054 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:45,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:45,058 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:45,058 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,058 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:45,058 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:45,059 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,059 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,060 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:45,061 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:45,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:45,062 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:45,062 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:45,062 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:45,062 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:45,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6807b0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:45,070 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:45,070 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,072 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:45,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:45,073 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,074 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:45,074 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:45,074 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:45,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:45,076 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:45,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:45,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:45,077 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:45,077 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:45,078 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,078 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:45,078 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,079 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,081 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:45,084 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:45,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:45,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:45,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:45,085 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:45,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:45,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:45,085 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:45,085 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:45,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175405086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175405098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,174 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/74db48047fd6494dacc3fd99211dd137 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74db48047fd6494dacc3fd99211dd137 2023-04-22 14:55:45,179 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 74db48047fd6494dacc3fd99211dd137(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:45,179 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:45,179 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=11, startTime=1682175344718; duration=0sec 2023-04-22 14:55:45,179 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:45,186 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:45,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:45,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:45,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:45,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:45,190 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:45,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:45,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:45,191 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,191 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,192 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:45,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:45,192 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:45,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:45,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:45,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:45,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:45,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:45,204 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:45,204 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:45,204 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:45,204 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:45,206 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,206 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,207 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:45,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a53611[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:45,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:45,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,211 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:45,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:45,211 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:45,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:45,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:45,219 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:45,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:45,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:45,219 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:45,219 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:45,221 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,222 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:45,222 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:45,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:45,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:45,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:45,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:45,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:45,229 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:45,229 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:45,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:45,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:45,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:45,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:45,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f15b1c8bc820453781d4cc5c6e3b7040 2023-04-22 14:55:45,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/5e5c924e58f24da89ceb13d5ac76f909 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909 2023-04-22 14:55:45,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909, entries=200, sequenceid=123, filesize=13.9 K 2023-04-22 14:55:45,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/cc93b097ca6a4d62a81d1d929d965ee3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3 2023-04-22 14:55:45,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3, entries=150, sequenceid=123, filesize=11.6 K 2023-04-22 14:55:45,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f15b1c8bc820453781d4cc5c6e3b7040 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040 2023-04-22 14:55:45,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040, entries=150, sequenceid=123, filesize=11.6 K 2023-04-22 14:55:45,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 95b9d53f66a350cbea7732aecb64c699 in 535ms, sequenceid=123, compaction requested=false 2023-04-22 14:55:45,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:45,329 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:45,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:45,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:45,334 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:45,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:45,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:45,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,336 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:45,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:45,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,336 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:45,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:45,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:45,345 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:45,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:45,346 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:45,346 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:45,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:45,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:45,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175345419, max timestamp=1682175345454Num uniques 859; 2023-04-22 14:55:45,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:45,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175345419, max timestamp=1682175345454Num uniques 859; 2023-04-22 14:55:45,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:45,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175405498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175405501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175405610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175405610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,771 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/ca56f48f6278465eb7ab06cef33664a2 2023-04-22 14:55:45,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175405816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175405816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175405848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,858 DEBUG [Thread-2873] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175405861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,869 DEBUG [Thread-2875] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:45,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:45,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175405870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:45,877 DEBUG [Thread-2869] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:46,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175406125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175406127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,187 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/57569e2449a0437696fc915db5a2f786 2023-04-22 14:55:46,198 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/910e472d92564acea0cd60f02667b538 2023-04-22 14:55:46,204 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/ca56f48f6278465eb7ab06cef33664a2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2 2023-04-22 14:55:46,207 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2, entries=150, sequenceid=135, filesize=11.7 K 2023-04-22 14:55:46,208 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/57569e2449a0437696fc915db5a2f786 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786 2023-04-22 14:55:46,214 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786, entries=150, sequenceid=135, filesize=11.7 K 2023-04-22 14:55:46,215 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/910e472d92564acea0cd60f02667b538 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538 2023-04-22 14:55:46,220 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538, entries=150, sequenceid=135, filesize=11.7 K 2023-04-22 14:55:46,221 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=136.42 KB/139690 for 95b9d53f66a350cbea7732aecb64c699 in 876ms, sequenceid=135, compaction requested=true 2023-04-22 14:55:46,221 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:46,221 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:46,226 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b652c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:46,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:46,227 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:46,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,229 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:46,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:46,230 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:46,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:46,234 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:46,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:46,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:46,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:46,234 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:46,237 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,237 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,237 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:46,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,241 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:46,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:46,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,243 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:46,243 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:46,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:46,343 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:46,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:46,359 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:46,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:46,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:46,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:46,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,361 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:46,361 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:46,361 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,361 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:46,363 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=136.42 KB heapSize=393.47 KB 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:46,363 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:46,378 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/3aa5c94675d945618ce616c5d4fa5203 2023-04-22 14:55:46,397 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/5df8c2c59c5d4f94bc25091e19db17eb 2023-04-22 14:55:46,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:46,640 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:46,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175406658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175406659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175406762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:46,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175406770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,810 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/758d21245f63409ca9ba39986245c3da 2023-04-22 14:55:46,815 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/3aa5c94675d945618ce616c5d4fa5203 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203 2023-04-22 14:55:46,818 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:55:46,819 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/5df8c2c59c5d4f94bc25091e19db17eb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb 2023-04-22 14:55:46,830 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:55:46,831 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/758d21245f63409ca9ba39986245c3da as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da 2023-04-22 14:55:46,838 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da, entries=150, sequenceid=167, filesize=11.8 K 2023-04-22 14:55:46,839 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~393.42 KB/402864, currentSize=40.25 KB/41220 for 95b9d53f66a350cbea7732aecb64c699 in 476ms, sequenceid=167, compaction requested=true 2023-04-22 14:55:46,839 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:46,839 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,841 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae32d2b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:46,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:46,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,846 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:46,846 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,846 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:46,847 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,847 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:46,848 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,848 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:46,852 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:46,853 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:46,853 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:46,853 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:46,853 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:46,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:46,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:46,855 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:46,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:46,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:46,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:46,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:46,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:46,860 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:46,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:46,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:46,860 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:46,860 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:46,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:46,960 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:46,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:46,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:46,964 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:46,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:46,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:46,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:46,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,967 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:46,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:46,967 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:46,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:46,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:46,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:46,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:46,969 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:46,970 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:46,970 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:46,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:46,982 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/aada44b0f7d645bcbcd093e2536626a9 2023-04-22 14:55:46,996 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7bbd023cf1ce4cf19dac8ac6f3424eb6 2023-04-22 14:55:47,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175347019, max timestamp=1682175347049Num uniques 859; 2023-04-22 14:55:47,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175347019, max timestamp=1682175347049Num uniques 859; 2023-04-22 14:55:47,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:47,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175347019, max timestamp=1682175347049Num uniques 859; 2023-04-22 14:55:47,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:47,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:47,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175407118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175407127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175407227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175407241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,405 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/eb46564aefb544929eaaf35e216596c4 2023-04-22 14:55:47,409 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/aada44b0f7d645bcbcd093e2536626a9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9 2023-04-22 14:55:47,414 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9, entries=150, sequenceid=176, filesize=11.8 K 2023-04-22 14:55:47,415 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7bbd023cf1ce4cf19dac8ac6f3424eb6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6 2023-04-22 14:55:47,418 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6, entries=150, sequenceid=176, filesize=11.8 K 2023-04-22 14:55:47,420 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/eb46564aefb544929eaaf35e216596c4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4 2023-04-22 14:55:47,427 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4, entries=150, sequenceid=176, filesize=11.8 K 2023-04-22 14:55:47,428 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 95b9d53f66a350cbea7732aecb64c699 in 458ms, sequenceid=176, compaction requested=true 2023-04-22 14:55:47,428 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:47,428 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:47,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:47,433 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311dfc40[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:47,436 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:47,437 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:47,438 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,438 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:47,441 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:47,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:47,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:47,446 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:47,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:47,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:47,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:47,446 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:47,449 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,449 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,449 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,451 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,453 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:47,463 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,463 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,463 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:47,463 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175407468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/6c578f429b034a9eae681d35644b84d9 2023-04-22 14:55:47,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175407475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,563 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:47,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:47,567 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:47,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:47,567 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:47,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:47,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:47,571 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,571 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:47,571 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:47,571 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:47,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b141cc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:47,575 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:47,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,577 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:47,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,577 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:47,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:47,581 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:47,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:47,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:47,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:47,581 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:47,583 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,584 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,585 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,586 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,589 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:47,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:47,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,589 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:47,589 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,589 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175407597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175407601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,689 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:47,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:47,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:47,693 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:47,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:47,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:47,694 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,694 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,695 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,695 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:47,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:47,698 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,698 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:47,698 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,698 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:47,698 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:47,698 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:47,700 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,700 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:47,702 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266d37f3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:47,702 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,704 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:47,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,704 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:47,708 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:47,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:47,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:47,710 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:47,710 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:47,711 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:47,711 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:47,711 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:47,713 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,713 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,713 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,715 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,717 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,719 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,719 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:47,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:47,720 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:47,720 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:47,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175407807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:47,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175407807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,820 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:47,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:47,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:47,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:47,824 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:47,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:47,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:47,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:47,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:47,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:47,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:47,831 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,831 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:47,831 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:47,831 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:47,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,835 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e4dc60[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:47,838 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:47,838 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:47,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,841 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:47,842 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:47,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,842 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:47,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:47,845 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:47,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:47,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:47,845 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:47,845 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:47,848 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,848 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,848 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,850 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,852 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,862 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,862 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:47,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:47,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:47,863 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,863 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,863 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:47,863 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:47,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/6ec7ac5c807a4042a6b7dac7cb5211a7 2023-04-22 14:55:47,963 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:47,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:47,968 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:47,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:47,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:47,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,969 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:47,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:47,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:47,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:47,973 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,973 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:47,973 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:47,973 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:47,975 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b45cb60[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:47,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:47,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,980 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:47,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:47,982 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:47,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:47,986 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:47,986 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:47,986 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:47,986 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:47,986 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:47,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,989 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:47,990 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,993 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:47,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:47,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:47,994 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:47,994 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:47,994 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:47,994 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:47,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:47,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,094 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:48,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:48,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:48,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:48,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:48,098 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:48,098 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,098 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:48,099 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:48,099 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,099 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,100 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:48,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:48,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:48,102 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,102 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:48,102 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:48,102 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:48,105 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,105 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,105 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,105 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79784ae5[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:48,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:48,107 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,109 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:48,111 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:48,111 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:48,111 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:48,113 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,113 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,113 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,114 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:48,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:48,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,119 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:48,119 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:48,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,120 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46830 deadline: 1682175408121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1682175408121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,220 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:48,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:48,223 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:48,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:48,223 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:48,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,224 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,225 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,225 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:48,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:48,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:48,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:48,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:48,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36653896[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:48,235 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:48,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:48,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:48,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:48,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:48,243 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:48,243 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:48,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,245 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:48,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:48,250 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:48,250 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/28cb3ea57db9472d860fd67a7f8904a5 2023-04-22 14:55:48,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/6c578f429b034a9eae681d35644b84d9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9 2023-04-22 14:55:48,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9, entries=200, sequenceid=213, filesize=14.1 K 2023-04-22 14:55:48,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/6ec7ac5c807a4042a6b7dac7cb5211a7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7 2023-04-22 14:55:48,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7, entries=150, sequenceid=213, filesize=11.8 K 2023-04-22 14:55:48,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/28cb3ea57db9472d860fd67a7f8904a5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5 2023-04-22 14:55:48,350 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:48,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:48,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:48,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5, entries=150, sequenceid=213, filesize=11.8 K 2023-04-22 14:55:48,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 95b9d53f66a350cbea7732aecb64c699 in 919ms, sequenceid=213, compaction requested=true 2023-04-22 14:55:48,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:48,363 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:48,363 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:48,363 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:48,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:48,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:48,364 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:48,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:48,366 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76768 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:48,366 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:48,366 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,366 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74db48047fd6494dacc3fd99211dd137, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=75.0 K 2023-04-22 14:55:48,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:48,367 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 74db48047fd6494dacc3fd99211dd137, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:48,367 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72038 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:48,367 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:48,367 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,367 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/be89f40483b842b4a9d90dfeddacc662, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=70.3 K 2023-04-22 14:55:48,368 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5e5c924e58f24da89ceb13d5ac76f909, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175344393 2023-04-22 14:55:48,368 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting be89f40483b842b4a9d90dfeddacc662, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:48,368 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ca56f48f6278465eb7ab06cef33664a2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175344754 2023-04-22 14:55:48,369 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3aa5c94675d945618ce616c5d4fa5203, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175345488 2023-04-22 14:55:48,369 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting cc93b097ca6a4d62a81d1d929d965ee3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175344393 2023-04-22 14:55:48,369 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting aada44b0f7d645bcbcd093e2536626a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682175346646 2023-04-22 14:55:48,369 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 57569e2449a0437696fc915db5a2f786, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175344754 2023-04-22 14:55:48,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,369 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,369 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 6c578f429b034a9eae681d35644b84d9, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347098 2023-04-22 14:55:48,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:48,370 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5df8c2c59c5d4f94bc25091e19db17eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175345488 2023-04-22 14:55:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,370 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7bbd023cf1ce4cf19dac8ac6f3424eb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682175346646 2023-04-22 14:55:48,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,371 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6ec7ac5c807a4042a6b7dac7cb5211a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347117 2023-04-22 14:55:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:48,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:48,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:48,389 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:48,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:48,392 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:48,392 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,392 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:48,392 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 14:55:48,393 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:48,393 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,393 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:48,394 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,394 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:48,394 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,394 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#408 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:48,424 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/d47c98d809e24bcd9306ab6ebc2e9a3e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d47c98d809e24bcd9306ab6ebc2e9a3e 2023-04-22 14:55:48,427 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/09aeab316cb94e7eb6365de96716a067 2023-04-22 14:55:48,430 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into d47c98d809e24bcd9306ab6ebc2e9a3e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:48,430 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:48,430 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=10, startTime=1682175348363; duration=0sec 2023-04-22 14:55:48,430 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:48,430 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:48,433 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72038 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:48,434 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:48,434 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,434 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d07292a8351c4207805fc67f428ccf4d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=70.3 K 2023-04-22 14:55:48,434 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d07292a8351c4207805fc67f428ccf4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682175344214 2023-04-22 14:55:48,435 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f15b1c8bc820453781d4cc5c6e3b7040, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682175344393 2023-04-22 14:55:48,436 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 910e472d92564acea0cd60f02667b538, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682175344754 2023-04-22 14:55:48,437 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 758d21245f63409ca9ba39986245c3da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682175345488 2023-04-22 14:55:48,437 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting eb46564aefb544929eaaf35e216596c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682175346646 2023-04-22 14:55:48,437 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 28cb3ea57db9472d860fd67a7f8904a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347117 2023-04-22 14:55:48,450 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:48,453 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/13f67f1085e242788f6308f70234ec3f 2023-04-22 14:55:48,475 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/dd2371d644db4497b346e1d011249a97 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dd2371d644db4497b346e1d011249a97 2023-04-22 14:55:48,481 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3411fc90f3be4f2c81d9ca34dd365619 2023-04-22 14:55:48,485 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into dd2371d644db4497b346e1d011249a97(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:48,485 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:48,485 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=10, startTime=1682175348367; duration=0sec 2023-04-22 14:55:48,485 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:48,488 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/09aeab316cb94e7eb6365de96716a067 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067 2023-04-22 14:55:48,492 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067, entries=100, sequenceid=221, filesize=9.4 K 2023-04-22 14:55:48,493 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/13f67f1085e242788f6308f70234ec3f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f 2023-04-22 14:55:48,498 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f, entries=100, sequenceid=221, filesize=9.4 K 2023-04-22 14:55:48,499 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3411fc90f3be4f2c81d9ca34dd365619 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619 2023-04-22 14:55:48,505 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619, entries=100, sequenceid=221, filesize=9.4 K 2023-04-22 14:55:48,506 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 114ms, sequenceid=221, compaction requested=false 2023-04-22 14:55:48,506 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:48,506 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:48,512 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:48,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e1f5e6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:48,521 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,521 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:48,523 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:48,526 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:48,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:48,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:48,526 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:48,526 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:48,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,529 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:48,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,533 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,538 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,538 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:48,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:48,538 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:48,539 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,539 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:48,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:48,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:48,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:48,539 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:48,539 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:48,639 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:48,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:48,644 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:48,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:48,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:48,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,646 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:48,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:48,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:48,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:48,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:48,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:48,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:48,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:48,648 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-22 14:55:48,648 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:48,649 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,649 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:48,649 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,649 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:48,649 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:48,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:48,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175348665, max timestamp=1682175348731Num uniques 859; 2023-04-22 14:55:48,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175348665, max timestamp=1682175348731Num uniques 859; 2023-04-22 14:55:48,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:48,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:48,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175348665, max timestamp=1682175348731Num uniques 859; 2023-04-22 14:55:48,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175348795, max timestamp=1682175348815Num uniques 1068; 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175348795, max timestamp=1682175348815Num uniques 1068; 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682175348795, max timestamp=1682175348815Num uniques 1018; 2023-04-22 14:55:48,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-22 14:55:48,826 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/0bf689dcabcf4989ac4547e2740bca85 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0bf689dcabcf4989ac4547e2740bca85 2023-04-22 14:55:48,833 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 0bf689dcabcf4989ac4547e2740bca85(size=12.1 K), total size for store is 21.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:48,833 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:48,833 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=10, startTime=1682175348364; duration=0sec 2023-04-22 14:55:48,834 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:48,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175348871, max timestamp=1682175348918Num uniques 1077; 2023-04-22 14:55:48,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175348871, max timestamp=1682175348918Num uniques 1077; 2023-04-22 14:55:48,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-22 14:55:48,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682175348871, max timestamp=1682175348918Num uniques 1077; 2023-04-22 14:55:48,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-22 14:55:48,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-22 14:55:48,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682175348946, max timestamp=1682175348982Num uniques 1086; 2023-04-22 14:55:48,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-22 14:55:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682175348946, max timestamp=1682175348982Num uniques 1086; 2023-04-22 14:55:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-22 14:55:48,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682175348946, max timestamp=1682175348982Num uniques 1086; 2023-04-22 14:55:48,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-22 14:55:49,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175349022, max timestamp=1682175349044Num uniques 1095; 2023-04-22 14:55:49,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-22 14:55:49,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175349022, max timestamp=1682175349044Num uniques 1095; 2023-04-22 14:55:49,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-22 14:55:49,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175349022, max timestamp=1682175349044Num uniques 1095; 2023-04-22 14:55:49,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-22 14:55:49,077 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/d9f13581cefa4a938e3413ce6356441c 2023-04-22 14:55:49,107 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d4fe610657c24677b42b8afa97615692 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175349073, max timestamp=1682175349125Num uniques 1054; 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175349073, max timestamp=1682175349125Num uniques 1054; 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682175349073, max timestamp=1682175349125Num uniques 1054; 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-22 14:55:49,128 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3a14762f1c474dd08d5652d99d9e945e 2023-04-22 14:55:49,139 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/d9f13581cefa4a938e3413ce6356441c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c 2023-04-22 14:55:49,144 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c, entries=200, sequenceid=231, filesize=14.1 K 2023-04-22 14:55:49,146 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d4fe610657c24677b42b8afa97615692 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692 2023-04-22 14:55:49,155 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692, entries=100, sequenceid=231, filesize=9.4 K 2023-04-22 14:55:49,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3a14762f1c474dd08d5652d99d9e945e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e 2023-04-22 14:55:49,160 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e, entries=100, sequenceid=231, filesize=9.4 K 2023-04-22 14:55:49,161 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=93.93 KB/96180 for 95b9d53f66a350cbea7732aecb64c699 in 513ms, sequenceid=231, compaction requested=true 2023-04-22 14:55:49,161 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:49,161 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,161 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:49,161 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:49,161 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:49,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,166 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=100.63 KB heapSize=281.16 KB 2023-04-22 14:55:49,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602a26f1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:49,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:49,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:49,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:49,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:49,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:49,169 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:49,169 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:49,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:49,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:49,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,175 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:49,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:49,178 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:49,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:49,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:49,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:49,178 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:49,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,182 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,183 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,186 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:49,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:49,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,187 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,187 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,187 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/4de492067a6141c2af8d63c9fff2bd3f 2023-04-22 14:55:49,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/5d1d79d7037140508121b6aeaff09341 2023-04-22 14:55:49,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175409225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175409229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,287 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:49,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:49,291 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:49,291 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,291 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:49,291 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:49,292 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,292 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,294 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,294 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,295 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,295 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:49,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:49,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:49,300 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,300 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:49,300 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:49,300 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:49,302 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbe7773[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:49,304 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:49,304 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,305 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:49,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:49,305 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:49,307 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:49,307 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,307 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:49,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:49,310 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:49,311 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:49,311 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:49,311 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:49,311 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:49,313 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,313 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,313 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,315 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,317 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:49,319 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,320 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,320 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,320 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,320 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,320 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:49,320 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:49,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175409330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175409349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,420 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:49,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:49,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:49,424 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:49,424 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,424 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:49,424 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:49,425 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,425 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,426 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:49,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:49,436 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,436 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:49,436 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:49,436 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,441 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4480551a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:49,443 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:49,443 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,445 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:49,445 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,445 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:49,446 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:49,449 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:49,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:49,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:49,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:49,449 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:49,452 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,452 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,452 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,454 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,455 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,458 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,458 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,459 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:49,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:49,459 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,459 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:49,459 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:49,459 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,460 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175409539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,559 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:49,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:49,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:49,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:49,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:49,562 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:49,562 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,563 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:49,563 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:49,563 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,563 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,564 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,564 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:49,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:49,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175409559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:49,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef4c6a9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:49,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:49,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:49,574 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,574 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:49,576 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:49,579 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:49,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:49,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:49,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:49,579 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:49,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:49,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:49,581 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,583 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,585 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:49,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:49,585 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:49,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:49,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:49,585 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:49,586 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:49,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e2ec5a0884bf46f9ba3b9305ee222122 2023-04-22 14:55:49,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/4de492067a6141c2af8d63c9fff2bd3f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f 2023-04-22 14:55:49,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f, entries=200, sequenceid=349, filesize=14.3 K 2023-04-22 14:55:49,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/5d1d79d7037140508121b6aeaff09341 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341 2023-04-22 14:55:49,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341, entries=150, sequenceid=349, filesize=11.9 K 2023-04-22 14:55:49,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e2ec5a0884bf46f9ba3b9305ee222122 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122 2023-04-22 14:55:49,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122, entries=150, sequenceid=349, filesize=11.9 K 2023-04-22 14:55:49,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~281.11 KB/287856, currentSize=80.51 KB/82440 for 95b9d53f66a350cbea7732aecb64c699 in 507ms, sequenceid=349, compaction requested=true 2023-04-22 14:55:49,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:49,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:49,675 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:49,675 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:49,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:49,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:49,677 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51102 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:49,677 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:49,677 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:49,677 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,677 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:49,677 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d47c98d809e24bcd9306ab6ebc2e9a3e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=49.9 K 2023-04-22 14:55:49,677 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,677 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0bf689dcabcf4989ac4547e2740bca85, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=42.9 K 2023-04-22 14:55:49,678 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d47c98d809e24bcd9306ab6ebc2e9a3e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347117 2023-04-22 14:55:49,678 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0bf689dcabcf4989ac4547e2740bca85, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347117 2023-04-22 14:55:49,678 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 09aeab316cb94e7eb6365de96716a067, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682175347467 2023-04-22 14:55:49,678 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 13f67f1085e242788f6308f70234ec3f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682175347467 2023-04-22 14:55:49,679 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d4fe610657c24677b42b8afa97615692, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175348631 2023-04-22 14:55:49,679 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d9f13581cefa4a938e3413ce6356441c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175348631 2023-04-22 14:55:49,679 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5d1d79d7037140508121b6aeaff09341, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:49,679 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 4de492067a6141c2af8d63c9fff2bd3f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:49,686 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:49,690 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#419 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:49,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:49,700 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:49,700 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,700 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:49,700 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:49,701 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,701 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,702 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#420 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:49,702 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,702 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:49,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:49,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:49,703 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,703 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:49,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:49,714 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,714 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:49,714 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:49,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:49,735 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/676b2cbb997a4aff8d642c92b810d771 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/676b2cbb997a4aff8d642c92b810d771 2023-04-22 14:55:49,741 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 676b2cbb997a4aff8d642c92b810d771(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:49,741 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:49,742 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=12, startTime=1682175349675; duration=0sec 2023-04-22 14:55:49,742 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:49,742 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:49,744 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:49,744 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:49,745 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:49,745 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dd2371d644db4497b346e1d011249a97, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=42.9 K 2023-04-22 14:55:49,745 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting dd2371d644db4497b346e1d011249a97, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682175347117 2023-04-22 14:55:49,745 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3411fc90f3be4f2c81d9ca34dd365619, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682175347467 2023-04-22 14:55:49,746 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3a14762f1c474dd08d5652d99d9e945e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682175348631 2023-04-22 14:55:49,746 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e2ec5a0884bf46f9ba3b9305ee222122, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:49,756 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/23bafacd1a124823acedd59c56742721 2023-04-22 14:55:49,758 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#422 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:49,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:49,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175409893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175409900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175409901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175409908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,915 DEBUG [Thread-2875] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175409915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:49,922 DEBUG [Thread-2869] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8219 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:50,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175410001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175410007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175410007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,127 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/6246327f3193468ba8c5646031996ed4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6246327f3193468ba8c5646031996ed4 2023-04-22 14:55:50,131 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 6246327f3193468ba8c5646031996ed4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:50,131 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:50,131 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=12, startTime=1682175349675; duration=0sec 2023-04-22 14:55:50,132 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:50,180 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/948997a843ed4e61945d8dd96a3fe8c5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/948997a843ed4e61945d8dd96a3fe8c5 2023-04-22 14:55:50,185 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into 948997a843ed4e61945d8dd96a3fe8c5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:50,185 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:50,185 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=12, startTime=1682175349676; duration=0sec 2023-04-22 14:55:50,185 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:50,188 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/b45d81dfa04b49199e97ea32c18bb667 2023-04-22 14:55:50,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175410208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175410215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175410215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175410517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175410520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:50,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175410523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,608 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/6ae3c1c282824f33a458abba24d59178 2023-04-22 14:55:50,612 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/23bafacd1a124823acedd59c56742721 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721 2023-04-22 14:55:50,615 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721, entries=150, sequenceid=364, filesize=11.9 K 2023-04-22 14:55:50,616 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/b45d81dfa04b49199e97ea32c18bb667 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667 2023-04-22 14:55:50,620 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667, entries=150, sequenceid=364, filesize=11.9 K 2023-04-22 14:55:50,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/6ae3c1c282824f33a458abba24d59178 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178 2023-04-22 14:55:50,624 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178, entries=150, sequenceid=364, filesize=11.9 K 2023-04-22 14:55:50,625 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 95b9d53f66a350cbea7732aecb64c699 in 911ms, sequenceid=364, compaction requested=false 2023-04-22 14:55:50,626 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:50,626 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:50,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f586a28[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:50,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:50,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:50,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:50,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:50,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:50,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:50,637 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:50,639 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:50,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:50,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:50,639 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:50,639 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:50,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,642 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:50,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:50,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:50,649 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:50,649 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:50,649 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:50,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:50,749 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:50,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:50,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:50,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:50,753 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:50,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:50,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:50,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:50,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:50,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:50,755 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:50,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:50,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:50,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:50,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:50,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:50,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:50,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:50,757 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:50,758 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:50,758 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:50,776 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/8b590197a05a44f1a6451e4d236ecaf7 2023-04-22 14:55:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:51,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:51,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175411042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175411043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175411047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175411150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175411150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175411150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,201 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/849bef887329497699dafbc44d317fba 2023-04-22 14:55:51,211 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/bed6c41dba984c7ba2df7a8778bc518c 2023-04-22 14:55:51,214 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/8b590197a05a44f1a6451e4d236ecaf7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7 2023-04-22 14:55:51,219 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7, entries=150, sequenceid=385, filesize=11.9 K 2023-04-22 14:55:51,220 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/849bef887329497699dafbc44d317fba as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba 2023-04-22 14:55:51,223 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba, entries=150, sequenceid=385, filesize=11.9 K 2023-04-22 14:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/bed6c41dba984c7ba2df7a8778bc518c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c 2023-04-22 14:55:51,229 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c, entries=150, sequenceid=385, filesize=11.9 K 2023-04-22 14:55:51,230 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 95b9d53f66a350cbea7732aecb64c699 in 472ms, sequenceid=385, compaction requested=true 2023-04-22 14:55:51,230 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:51,230 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f5895f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,234 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,235 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,235 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,235 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,240 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:51,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,242 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,242 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,242 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,244 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,249 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,250 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,250 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,350 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:51,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:51,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:51,354 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:51,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:51,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:51,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,354 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,356 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:51,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:51,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:51,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-22 14:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:51,359 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:51,359 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:51,359 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:51,359 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,363 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211b383e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,372 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,375 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,375 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,377 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,377 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/34b905f0c0cc445e84b8906482ec2ebe 2023-04-22 14:55:51,377 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,384 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,384 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:51,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175411394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175411400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175411401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,484 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:51,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:51,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:51,487 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:51,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:51,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:51,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,489 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,489 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:51,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:51,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:51,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:51,491 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:51,491 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,492 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7982733a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,495 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,499 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,499 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,505 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,506 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,506 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175411504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175411508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175411508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,606 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:51,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:51,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:51,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:51,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:51,610 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:51,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:51,610 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:51,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,612 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,612 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,612 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:51,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:51,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:51,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:51,614 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,614 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:51,614 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:51,614 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,617 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75075f08[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,621 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,625 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,625 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,634 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,640 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,641 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,641 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175411712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175411714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175411715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,741 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:51,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:51,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:51,744 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:51,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:51,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:51,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,745 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,746 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,747 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:51,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:51,748 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,748 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:51,749 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:51,749 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,750 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef38f86[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,754 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,755 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:51,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,758 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,758 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,762 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,765 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,765 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,765 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/bb120cb1b4704fe2b4de0921bea40d16 2023-04-22 14:55:51,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3aacfd492ece4d57a693bdb66bef3359 2023-04-22 14:55:51,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/34b905f0c0cc445e84b8906482ec2ebe as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe 2023-04-22 14:55:51,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe, entries=250, sequenceid=401, filesize=16.7 K 2023-04-22 14:55:51,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/bb120cb1b4704fe2b4de0921bea40d16 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16 2023-04-22 14:55:51,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16, entries=150, sequenceid=401, filesize=11.9 K 2023-04-22 14:55:51,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/3aacfd492ece4d57a693bdb66bef3359 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359 2023-04-22 14:55:51,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359, entries=150, sequenceid=401, filesize=11.9 K 2023-04-22 14:55:51,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 95b9d53f66a350cbea7732aecb64c699 in 460ms, sequenceid=401, compaction requested=true 2023-04-22 14:55:51,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:51,820 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:51,820 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:51,820 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:51,820 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:51,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:51,825 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:51,825 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:51,825 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,825 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6246327f3193468ba8c5646031996ed4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=48.1 K 2023-04-22 14:55:51,825 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:51,825 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:51,825 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6246327f3193468ba8c5646031996ed4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:51,826 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,826 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/676b2cbb997a4aff8d642c92b810d771, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=52.9 K 2023-04-22 14:55:51,826 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b45d81dfa04b49199e97ea32c18bb667, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682175349211 2023-04-22 14:55:51,826 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 676b2cbb997a4aff8d642c92b810d771, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:51,826 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 849bef887329497699dafbc44d317fba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682175349870 2023-04-22 14:55:51,826 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 23bafacd1a124823acedd59c56742721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682175349211 2023-04-22 14:55:51,827 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting bb120cb1b4704fe2b4de0921bea40d16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:51,827 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 8b590197a05a44f1a6451e4d236ecaf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682175349870 2023-04-22 14:55:51,827 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 34b905f0c0cc445e84b8906482ec2ebe, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:51,835 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:51,836 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:51,861 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/61edda45db5c47e3bd62ead6498ebe06 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/61edda45db5c47e3bd62ead6498ebe06 2023-04-22 14:55:51,865 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:51,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:51,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:51,870 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 61edda45db5c47e3bd62ead6498ebe06(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:51,870 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:51,870 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=12, startTime=1682175351820; duration=0sec 2023-04-22 14:55:51,870 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:51,870 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:51,872 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:51,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:51,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:51,873 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:51,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,873 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:51,873 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,873 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/948997a843ed4e61945d8dd96a3fe8c5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=48.1 K 2023-04-22 14:55:51,873 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 948997a843ed4e61945d8dd96a3fe8c5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682175349142 2023-04-22 14:55:51,874 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6ae3c1c282824f33a458abba24d59178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682175349211 2023-04-22 14:55:51,874 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting bed6c41dba984c7ba2df7a8778bc518c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682175349870 2023-04-22 14:55:51,875 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,875 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3aacfd492ece4d57a693bdb66bef3359, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:51,875 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,877 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:51,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:51,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:51,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:51,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,879 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:51,879 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:51,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:51,882 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:51,903 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/be3b6f1954f64fe1a040f81541fd36d1 2023-04-22 14:55:51,922 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/31079928ae6d4861a1209a0b70148f1c 2023-04-22 14:55:51,945 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/69828d7286b24c1bb3517bf9ad805452 2023-04-22 14:55:51,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/be3b6f1954f64fe1a040f81541fd36d1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1 2023-04-22 14:55:51,958 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 14:55:51,959 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/31079928ae6d4861a1209a0b70148f1c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c 2023-04-22 14:55:51,966 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 14:55:51,967 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/69828d7286b24c1bb3517bf9ad805452 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452 2023-04-22 14:55:51,973 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452, entries=150, sequenceid=418, filesize=11.9 K 2023-04-22 14:55:51,974 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 95ms, sequenceid=418, compaction requested=false 2023-04-22 14:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:51,976 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d18aae[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:51,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:51,978 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:51,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,980 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:51,981 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:51,983 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:51,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:51,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:51,983 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:51,984 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:51,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:51,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:51,992 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:51,996 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:51,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:51,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:51,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:51,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:51,997 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:51,997 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:51,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,997 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:51,997 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:52,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:52,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/5191926c54de4c39ba2fe74003392781 2023-04-22 14:55:52,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d919266a2f3a4e429f3a832560353721 2023-04-22 14:55:52,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/d89b4beaced84d768050e3832fd3bb91 2023-04-22 14:55:52,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/5191926c54de4c39ba2fe74003392781 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781 2023-04-22 14:55:52,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781, entries=200, sequenceid=428, filesize=14.3 K 2023-04-22 14:55:52,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d919266a2f3a4e429f3a832560353721 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721 2023-04-22 14:55:52,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721, entries=150, sequenceid=428, filesize=11.9 K 2023-04-22 14:55:52,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/d89b4beaced84d768050e3832fd3bb91 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91 2023-04-22 14:55:52,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91, entries=150, sequenceid=428, filesize=11.9 K 2023-04-22 14:55:52,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for 95b9d53f66a350cbea7732aecb64c699 in 55ms, sequenceid=428, compaction requested=true 2023-04-22 14:55:52,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:52,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 14:55:52,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-22 14:55:52,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:52,097 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-22 14:55:52,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:52,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:52,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:52,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:52,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,102 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,103 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,104 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,105 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,106 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,106 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,106 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,107 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,108 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca03aed[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,117 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,118 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,119 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,123 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,123 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,123 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,125 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,127 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,131 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,132 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,133 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,133 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175412133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175412133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175412139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,233 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,236 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,236 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,238 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,239 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,240 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,241 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,241 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,242 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,243 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,243 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6390df6e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,248 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175412247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175412247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175412248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,252 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,254 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,255 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,256 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,259 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,259 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,259 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,260 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,261 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,261 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,261 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,267 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/e3f2c42957164dc48859d13533d31297 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e3f2c42957164dc48859d13533d31297 2023-04-22 14:55:52,272 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into e3f2c42957164dc48859d13533d31297(size=12.5 K), total size for store is 38.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:52,273 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,273 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=12, startTime=1682175351820; duration=0sec 2023-04-22 14:55:52,273 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-22 14:55:52,273 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-22 14:55:52,274 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 14:55:52,274 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 14:55:52,274 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:55:52,274 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:52,276 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:52,276 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:52,276 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,276 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e3f2c42957164dc48859d13533d31297, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=38.7 K 2023-04-22 14:55:52,277 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e3f2c42957164dc48859d13533d31297, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:52,277 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting be3b6f1954f64fe1a040f81541fd36d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682175351392 2023-04-22 14:55:52,277 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 5191926c54de4c39ba2fe74003392781, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682175352023 2023-04-22 14:55:52,283 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:52,300 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/d55e06338ad84835b5bec5bf435c473d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d55e06338ad84835b5bec5bf435c473d 2023-04-22 14:55:52,304 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into d55e06338ad84835b5bec5bf435c473d(size=12.5 K), total size for store is 36.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:52,304 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,304 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=12, startTime=1682175351824; duration=0sec 2023-04-22 14:55:52,304 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:52,304 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:52,309 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:52,309 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:52,309 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,309 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/61edda45db5c47e3bd62ead6498ebe06, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=36.3 K 2023-04-22 14:55:52,310 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 61edda45db5c47e3bd62ead6498ebe06, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:52,310 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 31079928ae6d4861a1209a0b70148f1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682175351392 2023-04-22 14:55:52,310 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d919266a2f3a4e429f3a832560353721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682175352023 2023-04-22 14:55:52,318 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:52,361 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,364 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,364 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,366 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,368 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,369 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,371 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413df325[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,375 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,376 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,379 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,379 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,381 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,383 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,386 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,386 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,392 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,392 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175412451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175412452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175412453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,492 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,496 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,498 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,499 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,504 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b739556[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,511 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,511 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,514 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,514 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,514 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/fe2040a17e144e05adfa6ebc15b5d1a8 2023-04-22 14:55:52,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,520 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,521 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,526 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,527 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,527 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,528 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,528 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/3a29b6865c2c4a15868d8723e902026c 2023-04-22 14:55:52,628 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,631 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,638 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,639 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,639 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,639 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,640 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,640 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2629b187[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,643 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,645 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,645 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,645 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,647 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,647 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,647 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,650 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,650 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,650 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,651 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,652 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,655 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,655 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,655 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,699 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/bb07cd9a6ba6402c8383a9988c3f859f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/bb07cd9a6ba6402c8383a9988c3f859f 2023-04-22 14:55:52,705 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into bb07cd9a6ba6402c8383a9988c3f859f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:52,705 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,705 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=13, startTime=1682175352090; duration=0sec 2023-04-22 14:55:52,706 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:52,726 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/b8f96ead33154147a732e59a872e31d8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b8f96ead33154147a732e59a872e31d8 2023-04-22 14:55:52,730 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into b8f96ead33154147a732e59a872e31d8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:52,730 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,730 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=13, startTime=1682175352091; duration=0sec 2023-04-22 14:55:52,730 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:52,755 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,759 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175412757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175412757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175412757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,761 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,763 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43277183[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,766 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,771 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,772 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,773 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,775 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,775 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,775 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,777 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,777 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,779 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,781 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,781 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,781 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,781 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,782 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,782 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,782 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,882 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:52,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:52,886 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:52,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:52,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:52,886 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,892 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:52,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:52,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:52,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:52,895 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dad0619[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:52,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:52,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,902 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:52,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:52,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:52,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,904 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:52,904 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:52,908 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:52,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:52,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:52,908 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:52,909 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:52,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,912 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:52,912 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:52,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:52,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:52,919 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:52,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:52,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:52,918 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:52,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:52,919 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:52,919 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:52,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:52,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e5bac47f38d44561a75dd5f218bc3d33 2023-04-22 14:55:52,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/fe2040a17e144e05adfa6ebc15b5d1a8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8 2023-04-22 14:55:52,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8, entries=300, sequenceid=447, filesize=19.0 K 2023-04-22 14:55:52,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/3a29b6865c2c4a15868d8723e902026c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c 2023-04-22 14:55:52,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c, entries=150, sequenceid=447, filesize=11.9 K 2023-04-22 14:55:52,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e5bac47f38d44561a75dd5f218bc3d33 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33 2023-04-22 14:55:52,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33, entries=150, sequenceid=447, filesize=11.9 K 2023-04-22 14:55:52,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 95b9d53f66a350cbea7732aecb64c699 in 893ms, sequenceid=447, compaction requested=true 2023-04-22 14:55:52,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:52,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:52,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:52,991 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 14:55:52,991 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-22 14:55:52,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:55:52,992 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:52,993 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:52,993 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:52,993 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:52,993 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d55e06338ad84835b5bec5bf435c473d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=48.2 K 2023-04-22 14:55:52,994 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d55e06338ad84835b5bec5bf435c473d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682175351033 2023-04-22 14:55:52,994 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 69828d7286b24c1bb3517bf9ad805452, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682175351392 2023-04-22 14:55:52,994 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d89b4beaced84d768050e3832fd3bb91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682175352023 2023-04-22 14:55:52,995 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e5bac47f38d44561a75dd5f218bc3d33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682175352077 2023-04-22 14:55:53,006 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#445 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:53,019 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:53,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:53,035 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:53,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:53,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:53,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,036 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:53,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:53,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:53,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:53,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:53,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:53,044 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:55:53,044 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,062 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/d19487347d3a4b968bd9bd600def3fdd 2023-04-22 14:55:53,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:53,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:53,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175413295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175413304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175413306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175413407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175413413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175413415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,438 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/47994eebb9624843861836cf2eb05c89 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/47994eebb9624843861836cf2eb05c89 2023-04-22 14:55:53,442 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into 47994eebb9624843861836cf2eb05c89(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:53,442 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:53,442 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=12, startTime=1682175352991; duration=0sec 2023-04-22 14:55:53,442 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:53,478 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d6477fc33db24fdb949a0700b930472e 2023-04-22 14:55:53,497 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/9e3f6180217b4c129a567827df4e0040 2023-04-22 14:55:53,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/d19487347d3a4b968bd9bd600def3fdd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd 2023-04-22 14:55:53,506 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd, entries=150, sequenceid=464, filesize=11.9 K 2023-04-22 14:55:53,507 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/d6477fc33db24fdb949a0700b930472e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e 2023-04-22 14:55:53,519 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e, entries=150, sequenceid=464, filesize=11.9 K 2023-04-22 14:55:53,520 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/9e3f6180217b4c129a567827df4e0040 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040 2023-04-22 14:55:53,525 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040, entries=150, sequenceid=464, filesize=11.9 K 2023-04-22 14:55:53,526 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 95b9d53f66a350cbea7732aecb64c699 in 482ms, sequenceid=464, compaction requested=true 2023-04-22 14:55:53,526 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:53,526 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:53,531 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae3f7df[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:53,534 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:53,534 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,535 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,537 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,537 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:53,537 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:53,541 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:53,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:53,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:53,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:53,541 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:53,542 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,543 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,543 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,543 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,545 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,551 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:53,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:53,551 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:53,551 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,551 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:53,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,551 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,551 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,551 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,552 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:53,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:53,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:53,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:53,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/7494657b0b53430a8dae5a79995d59cd 2023-04-22 14:55:53,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/58caa6a0e31c4ee0aacb2d5938b93841 2023-04-22 14:55:53,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175413639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175413639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175413640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,651 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:53,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:53,655 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:53,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:53,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:53,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,655 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,656 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,656 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,657 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:53,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:53,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:53,658 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,658 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:53,658 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:53,658 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,659 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:53,659 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:53,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,672 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,672 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112dc48d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:53,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:53,674 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,674 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:53,677 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:53,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:53,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:53,678 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:53,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:53,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:53,678 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:53,678 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:53,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,679 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,681 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,684 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,684 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:53,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,684 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,685 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:53,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:53,685 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:53,685 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:53,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175413750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175413750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175413751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,785 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:53,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:53,788 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:53,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:53,788 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:53,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,790 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:53,791 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:53,792 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,792 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:53,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:53,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:53,792 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:53,797 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fe1442[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:53,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:53,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,799 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:53,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,800 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:53,801 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:53,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:53,804 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:53,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:53,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:53,804 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:53,804 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:53,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,805 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,806 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,807 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:53,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,811 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:53,812 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:53,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,812 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,912 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:53,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:53,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:53,915 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:53,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:53,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:53,916 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,916 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,917 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,918 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:53,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:53,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:53,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:53,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:53,919 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:53,921 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c665aaa[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:53,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:53,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,924 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:53,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:53,930 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:53,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:53,933 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:53,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:53,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:53,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:53,933 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:53,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:53,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:53,936 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,938 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:53,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:53,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:53,940 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:53,941 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:53,941 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:53,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175413954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175413954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:53,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175413957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,041 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,044 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,046 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e0b7efff8f974ffbae5ed344e877d322 2023-04-22 14:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9de15f6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,055 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,057 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/7494657b0b53430a8dae5a79995d59cd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd 2023-04-22 14:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd, entries=200, sequenceid=485, filesize=14.3 K 2023-04-22 14:55:54,062 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,062 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/58caa6a0e31c4ee0aacb2d5938b93841 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841 2023-04-22 14:55:54,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,065 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,067 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841, entries=150, sequenceid=485, filesize=11.9 K 2023-04-22 14:55:54,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e0b7efff8f974ffbae5ed344e877d322 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:54,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,072 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,072 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322, entries=150, sequenceid=485, filesize=11.9 K 2023-04-22 14:55:54,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 95b9d53f66a350cbea7732aecb64c699 in 458ms, sequenceid=485, compaction requested=true 2023-04-22 14:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:54,075 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:54,075 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:54,076 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:54,076 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:54,076 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:55:54,076 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,076 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:54,076 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/bb07cd9a6ba6402c8383a9988c3f859f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=57.9 K 2023-04-22 14:55:54,076 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,077 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b8f96ead33154147a732e59a872e31d8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=48.3 K 2023-04-22 14:55:54,077 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting bb07cd9a6ba6402c8383a9988c3f859f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682175352023 2023-04-22 14:55:54,077 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b8f96ead33154147a732e59a872e31d8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682175352023 2023-04-22 14:55:54,077 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fe2040a17e144e05adfa6ebc15b5d1a8, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682175352075 2023-04-22 14:55:54,077 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3a29b6865c2c4a15868d8723e902026c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682175352077 2023-04-22 14:55:54,078 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d6477fc33db24fdb949a0700b930472e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682175352130 2023-04-22 14:55:54,078 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d19487347d3a4b968bd9bd600def3fdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682175352130 2023-04-22 14:55:54,078 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 58caa6a0e31c4ee0aacb2d5938b93841, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353302 2023-04-22 14:55:54,078 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7494657b0b53430a8dae5a79995d59cd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353293 2023-04-22 14:55:54,090 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:54,090 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:54,112 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/a91c2afed9834317b39cb0e30105d947 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a91c2afed9834317b39cb0e30105d947 2023-04-22 14:55:54,114 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/c1f771e2c0ca4bf08b0c7f0f88791cb5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/c1f771e2c0ca4bf08b0c7f0f88791cb5 2023-04-22 14:55:54,119 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into a91c2afed9834317b39cb0e30105d947(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:54,119 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:54,120 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=12, startTime=1682175354075; duration=0sec 2023-04-22 14:55:54,120 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:54,120 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:54,120 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into c1f771e2c0ca4bf08b0c7f0f88791cb5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:54,121 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:54,121 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=12, startTime=1682175354075; duration=0sec 2023-04-22 14:55:54,121 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:54,121 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:54,121 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:54,121 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,121 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/47994eebb9624843861836cf2eb05c89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=36.5 K 2023-04-22 14:55:54,121 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 47994eebb9624843861836cf2eb05c89, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682175352077 2023-04-22 14:55:54,122 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9e3f6180217b4c129a567827df4e0040, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682175352130 2023-04-22 14:55:54,122 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e0b7efff8f974ffbae5ed344e877d322, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353302 2023-04-22 14:55:54,127 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:54,145 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/a3254f20e9134017b5c188271b2b638c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a3254f20e9134017b5c188271b2b638c 2023-04-22 14:55:54,150 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into a3254f20e9134017b5c188271b2b638c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:54,150 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:54,150 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=13, startTime=1682175354075; duration=0sec 2023-04-22 14:55:54,150 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:54,172 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,175 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,176 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,176 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,177 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,177 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,177 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,179 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,179 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,179 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:54,180 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,192 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/ae4ee425efba48bcb9c1df130e67b6ee 2023-04-22 14:55:54,202 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/74afd1a2bcd44a8a9364a77539ab9e17 2023-04-22 14:55:54,211 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/09c4b08ce2e246c4b17f28d052a7cfd9 2023-04-22 14:55:54,216 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/ae4ee425efba48bcb9c1df130e67b6ee as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee 2023-04-22 14:55:54,222 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee, entries=150, sequenceid=500, filesize=11.9 K 2023-04-22 14:55:54,223 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/74afd1a2bcd44a8a9364a77539ab9e17 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17 2023-04-22 14:55:54,226 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17, entries=150, sequenceid=500, filesize=11.9 K 2023-04-22 14:55:54,227 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/09c4b08ce2e246c4b17f28d052a7cfd9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9 2023-04-22 14:55:54,231 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9, entries=150, sequenceid=500, filesize=11.9 K 2023-04-22 14:55:54,232 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 53ms, sequenceid=500, compaction requested=false 2023-04-22 14:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:54,232 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,235 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,235 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7628b1ec[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,237 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,237 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,238 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,239 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,242 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,245 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,245 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,247 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,247 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,252 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,252 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,252 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,253 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:54,254 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,254 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:54,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:54,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:54,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175354296, max timestamp=1682175354312Num uniques 859; 2023-04-22 14:55:54,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:54,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175354296, max timestamp=1682175354312Num uniques 859; 2023-04-22 14:55:54,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:54,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175354296, max timestamp=1682175354312Num uniques 859; 2023-04-22 14:55:54,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:54,354 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,357 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,358 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,359 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,359 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,360 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,361 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,361 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,361 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,361 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,363 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6df76f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,365 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,366 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,368 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175414363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175414363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,372 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,372 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,372 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,378 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,378 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,378 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175414370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,380 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,382 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,382 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,383 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,383 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175414472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175414472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175414479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,483 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,486 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,487 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,488 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,490 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,492 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31005322[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,493 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,497 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,500 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,500 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,502 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,504 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:54,507 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,507 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,607 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,611 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,613 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,613 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,615 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,617 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48888151[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,620 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,620 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,621 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,624 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,625 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,632 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,632 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/81e6784eb36d4faab34bec86e1b3773e 2023-04-22 14:55:54,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175414675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175414675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7f8e5741459e45b0b0ffa423c9457103 2023-04-22 14:55:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175414684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,732 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,735 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,736 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,736 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,736 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,736 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,736 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,737 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,738 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,738 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,739 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,739 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,739 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,739 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,741 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9cee7d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,744 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,745 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,747 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,747 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,750 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,750 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,751 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,752 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,759 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,760 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,760 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,860 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,863 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,863 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,864 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,865 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,865 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,867 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,868 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bcab8ee[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,873 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:54,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,873 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:54,876 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:54,879 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:54,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:54,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:54,879 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:54,879 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:54,881 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,881 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,881 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,881 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,882 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:54,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,885 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,885 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:54,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:54,885 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:54,885 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:54,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:54,985 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:54,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:54,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:54,989 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:54,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:54,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:54,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175414986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175414986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:54,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:54,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:54,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:54,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:54,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:54,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:54,993 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,993 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:54,993 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:54,993 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:54,996 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:54,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:54,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48607df6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:54,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:54,998 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,000 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,000 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46840 deadline: 1682175414997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,003 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,005 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,006 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,008 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,009 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,010 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,014 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,014 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,014 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f2d5be944a96498388eb1ed347bba178 2023-04-22 14:55:55,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/81e6784eb36d4faab34bec86e1b3773e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e 2023-04-22 14:55:55,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e, entries=250, sequenceid=510, filesize=16.7 K 2023-04-22 14:55:55,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7f8e5741459e45b0b0ffa423c9457103 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103 2023-04-22 14:55:55,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103, entries=150, sequenceid=510, filesize=11.9 K 2023-04-22 14:55:55,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f2d5be944a96498388eb1ed347bba178 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178 2023-04-22 14:55:55,115 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178, entries=150, sequenceid=510, filesize=11.9 K 2023-04-22 14:55:55,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,116 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 849ms, sequenceid=510, compaction requested=true 2023-04-22 14:55:55,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:55,116 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:55,116 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:55,117 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42324 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:55,117 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:55,117 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,117 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/c1f771e2c0ca4bf08b0c7f0f88791cb5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=41.3 K 2023-04-22 14:55:55,118 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,118 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:55,118 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,118 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a91c2afed9834317b39cb0e30105d947, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=36.6 K 2023-04-22 14:55:55,118 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c1f771e2c0ca4bf08b0c7f0f88791cb5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353302 2023-04-22 14:55:55,118 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a91c2afed9834317b39cb0e30105d947, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353302 2023-04-22 14:55:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,119 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 74afd1a2bcd44a8a9364a77539ab9e17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682175353630 2023-04-22 14:55:55,119 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,119 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ae4ee425efba48bcb9c1df130e67b6ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682175353630 2023-04-22 14:55:55,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,119 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,119 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7f8e5741459e45b0b0ffa423c9457103, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:55,119 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 81e6784eb36d4faab34bec86e1b3773e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:55,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,120 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,121 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,121 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,125 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-22 14:55:55,125 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:55,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:55,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:55,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:55,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:55,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:55,127 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#461 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:55,133 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:55,153 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/43b7d615f9584234803299212166ad1b 2023-04-22 14:55:55,158 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/1c26375dab0248d09913debf45c72ddc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/1c26375dab0248d09913debf45c72ddc 2023-04-22 14:55:55,165 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 1c26375dab0248d09913debf45c72ddc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:55,165 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,165 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=13, startTime=1682175355116; duration=0sec 2023-04-22 14:55:55,165 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:55,165 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:55,167 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:55,167 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:55,167 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,167 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a3254f20e9134017b5c188271b2b638c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=36.6 K 2023-04-22 14:55:55,171 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a3254f20e9134017b5c188271b2b638c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682175353302 2023-04-22 14:55:55,172 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 09c4b08ce2e246c4b17f28d052a7cfd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682175353630 2023-04-22 14:55:55,172 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/2e714c6fa6de4293b107ed6ab9bcdbee 2023-04-22 14:55:55,172 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f2d5be944a96498388eb1ed347bba178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:55,178 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:55,194 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/dca67e990a6a4d3c930304a69d84b29c 2023-04-22 14:55:55,198 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/43b7d615f9584234803299212166ad1b as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b 2023-04-22 14:55:55,204 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b, entries=150, sequenceid=545, filesize=11.9 K 2023-04-22 14:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/2e714c6fa6de4293b107ed6ab9bcdbee as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee 2023-04-22 14:55:55,209 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee, entries=150, sequenceid=545, filesize=11.9 K 2023-04-22 14:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/dca67e990a6a4d3c930304a69d84b29c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c 2023-04-22 14:55:55,213 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c, entries=150, sequenceid=545, filesize=11.9 K 2023-04-22 14:55:55,214 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 89ms, sequenceid=545, compaction requested=false 2023-04-22 14:55:55,214 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,214 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446eefa2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,218 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,218 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,220 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,221 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,223 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,226 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,226 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,226 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,228 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,228 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,229 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,230 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,233 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,232 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,233 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,233 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,233 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,333 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,338 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,342 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,342 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,343 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,345 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,345 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,348 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581d0e54[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,351 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,352 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,352 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,352 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,354 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,354 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,356 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,356 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,356 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,369 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:55,373 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,373 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,373 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,373 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,473 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,477 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,477 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,478 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,479 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,480 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,482 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,482 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,482 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,482 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,484 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bed071[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,486 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,487 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,488 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,490 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,490 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,494 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,494 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,498 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,500 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,501 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,501 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,501 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,501 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,501 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,501 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:55,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:55,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:55,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:55,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/a7c49addf60e448aae151502bad56c04 2023-04-22 14:55:55,552 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/87b265d497464e46b12b3b8399af5a50 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/87b265d497464e46b12b3b8399af5a50 2023-04-22 14:55:55,556 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 87b265d497464e46b12b3b8399af5a50(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:55,556 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,557 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=13, startTime=1682175355116; duration=0sec 2023-04-22 14:55:55,557 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:55,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175355553, max timestamp=1682175355565Num uniques 859; 2023-04-22 14:55:55,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175355553, max timestamp=1682175355565Num uniques 859; 2023-04-22 14:55:55,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:55,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175355553, max timestamp=1682175355565Num uniques 859; 2023-04-22 14:55:55,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:55,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:55,591 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/079973d17e55423fbf68992a7f769b61 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/079973d17e55423fbf68992a7f769b61 2023-04-22 14:55:55,594 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into 079973d17e55423fbf68992a7f769b61(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:55,594 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,594 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=13, startTime=1682175355116; duration=0sec 2023-04-22 14:55:55,594 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:55,601 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175415597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,605 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,606 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,607 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,607 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,608 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:55,609 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175415603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,612 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,612 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175415604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dfa9c9d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,615 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,616 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,616 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,617 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,617 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,627 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,627 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,629 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,631 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,636 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,636 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175415705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175415712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175415715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,736 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,743 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:55,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,749 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eda677f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,754 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,755 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,755 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,758 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,758 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,758 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,761 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,763 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,763 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,767 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,767 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,768 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,868 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:55,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:55,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:55,871 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:55,871 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,871 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:55,871 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:55,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,872 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,874 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,874 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:55,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:55,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:55,876 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,876 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:55,876 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:55,876 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:55,880 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e85b75e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:55,882 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:55,882 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,883 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:55,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:55,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:55,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:55,887 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:55,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:55,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:55,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:55,887 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:55,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:55,890 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:55,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:55,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:55,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:55,896 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:55,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:55,896 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:55,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:55,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175415913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175415920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175415926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:55,942 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/3be868c385364a2799d350c7feb74d25 2023-04-22 14:55:55,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f180baa212774621b9f7ef7a5ce1df60 2023-04-22 14:55:55,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/a7c49addf60e448aae151502bad56c04 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04 2023-04-22 14:55:55,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04, entries=150, sequenceid=558, filesize=11.9 K 2023-04-22 14:55:55,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/3be868c385364a2799d350c7feb74d25 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25 2023-04-22 14:55:55,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25, entries=100, sequenceid=558, filesize=9.5 K 2023-04-22 14:55:55,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f180baa212774621b9f7ef7a5ce1df60 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60 2023-04-22 14:55:55,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60, entries=100, sequenceid=558, filesize=9.5 K 2023-04-22 14:55:55,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 456ms, sequenceid=558, compaction requested=true 2023-04-22 14:55:55,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:55,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:55,967 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:55,967 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:55,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:55,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:55,967 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37542 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:55,968 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:55,968 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,968 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/87b265d497464e46b12b3b8399af5a50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=36.7 K 2023-04-22 14:55:55,968 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:55,968 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:55,968 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:55,968 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/1c26375dab0248d09913debf45c72ddc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=34.3 K 2023-04-22 14:55:55,969 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 87b265d497464e46b12b3b8399af5a50, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:55,969 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1c26375dab0248d09913debf45c72ddc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 43b7d615f9584234803299212166ad1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682175354356 2023-04-22 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2e714c6fa6de4293b107ed6ab9bcdbee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682175354356 2023-04-22 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting a7c49addf60e448aae151502bad56c04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175355502 2023-04-22 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3be868c385364a2799d350c7feb74d25, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175355502 2023-04-22 14:55:55,976 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#470 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:55,985 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:55,996 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:55,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:55,998 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/0aaec5506fd24dff9a00f591e15890ac as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/0aaec5506fd24dff9a00f591e15890ac 2023-04-22 14:55:55,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:56,004 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 0aaec5506fd24dff9a00f591e15890ac(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:56,004 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:56,005 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=13, startTime=1682175355966; duration=0sec 2023-04-22 14:55:56,005 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:56,005 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:55:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:56,005 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:56,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:56,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:56,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:56,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:56,007 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:55:56,007 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:56,007 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:56,007 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/079973d17e55423fbf68992a7f769b61, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=34.3 K 2023-04-22 14:55:56,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:56,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,008 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 079973d17e55423fbf68992a7f769b61, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682175354261 2023-04-22 14:55:56,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,008 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:56,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:56,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:56,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:56,013 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:56,016 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting dca67e990a6a4d3c930304a69d84b29c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682175354356 2023-04-22 14:55:56,018 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f180baa212774621b9f7ef7a5ce1df60, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175355502 2023-04-22 14:55:56,022 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/5aad955468884025847b23db50b1e94c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5aad955468884025847b23db50b1e94c 2023-04-22 14:55:56,039 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 5aad955468884025847b23db50b1e94c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:56,039 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:56,039 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=13, startTime=1682175355967; duration=0sec 2023-04-22 14:55:56,039 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:56,041 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:56,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:56,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:56,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175416240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175416242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175416243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175416346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175416347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175416352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,464 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/21ceea3f1d15494598f42e3eef11d8f2 2023-04-22 14:55:56,470 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/496b937aa3b54bc69999ff97f56f45c5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/496b937aa3b54bc69999ff97f56f45c5 2023-04-22 14:55:56,477 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into 496b937aa3b54bc69999ff97f56f45c5(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:56,477 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:56,477 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=13, startTime=1682175355967; duration=0sec 2023-04-22 14:55:56,477 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:56,479 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7a213519f7774e24a262b5b49202038a 2023-04-22 14:55:56,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175416550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175416553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175416558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175416856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175416859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175416863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,904 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/171dba8bf3594895a27a969367507f8c 2023-04-22 14:55:56,908 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/21ceea3f1d15494598f42e3eef11d8f2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2 2023-04-22 14:55:56,913 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2, entries=150, sequenceid=597, filesize=11.9 K 2023-04-22 14:55:56,914 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/7a213519f7774e24a262b5b49202038a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a 2023-04-22 14:55:56,918 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a, entries=150, sequenceid=597, filesize=11.9 K 2023-04-22 14:55:56,919 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/171dba8bf3594895a27a969367507f8c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c 2023-04-22 14:55:56,927 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c, entries=150, sequenceid=597, filesize=11.9 K 2023-04-22 14:55:56,928 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 95b9d53f66a350cbea7732aecb64c699 in 916ms, sequenceid=597, compaction requested=false 2023-04-22 14:55:56,928 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:56,929 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:56,930 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e38c5a7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:56,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:56,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,935 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:56,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:56,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:56,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,935 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:56,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:56,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:56,938 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,938 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:56,941 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:56,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:56,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:56,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:56,941 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:56,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:56,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:56,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:56,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:56,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:56,948 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:56,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:56,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:56,948 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:56,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:56,949 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:56,949 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:57,049 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:57,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:57,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:57,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:57,053 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:57,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:57,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:57,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:57,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:57,054 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:57,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:57,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,055 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:57,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:57,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:57,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:57,056 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:57,056 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:57,056 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:57,057 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:57,065 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/b1201f4b73bc467a997e486d27547ff7 2023-04-22 14:55:57,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:57,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175357389, max timestamp=1682175357400Num uniques 859; 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175357389, max timestamp=1682175357400Num uniques 859; 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175357389, max timestamp=1682175357400Num uniques 859; 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:57,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:55:57,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175417429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175417434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175417436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,477 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/9270cf34a9274ab781bb799de398e70e 2023-04-22 14:55:57,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175417538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175417538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175417545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175417745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175417746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:57,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175417749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,887 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/32f7820f553f4928b1ff5904368d6a1a 2023-04-22 14:55:57,891 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/b1201f4b73bc467a997e486d27547ff7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7 2023-04-22 14:55:57,895 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7, entries=150, sequenceid=609, filesize=11.9 K 2023-04-22 14:55:57,896 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/9270cf34a9274ab781bb799de398e70e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e 2023-04-22 14:55:57,900 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e, entries=150, sequenceid=609, filesize=11.9 K 2023-04-22 14:55:57,900 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/32f7820f553f4928b1ff5904368d6a1a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a 2023-04-22 14:55:57,903 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a, entries=150, sequenceid=609, filesize=11.9 K 2023-04-22 14:55:57,904 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 848ms, sequenceid=609, compaction requested=true 2023-04-22 14:55:57,904 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:57,904 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:57,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:57,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:57,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:57,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:57,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574fa889[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:57,909 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:57,909 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:57,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:57,915 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:57,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:57,917 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:57,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:57,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:57,917 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:57,917 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:57,920 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,920 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,920 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:57,921 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,922 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:57,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:57,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:57,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:57,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:57,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:57,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:57,926 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:57,926 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:57,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:58,026 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:58,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:58,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:58,030 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:58,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:58,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:58,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:58,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:58,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:58,031 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:58,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:58,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:58,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:58,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:58,033 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:58,034 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-22 14:55:58,034 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:58,034 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:58,034 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:58,034 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:58,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:58,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:58,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:58,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175418068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175418069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175418070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175418181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175418181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175418182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175418387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175418387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175418389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,446 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/eee65b65976347b2980e1b90e28d7829 2023-04-22 14:55:58,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175418697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175418697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175418698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,857 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/62fbc64c377241489a663af01ca2529d 2023-04-22 14:55:58,873 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e3bd0e45a5444ced9f80cc8539a833aa 2023-04-22 14:55:58,876 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/eee65b65976347b2980e1b90e28d7829 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829 2023-04-22 14:55:58,879 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 14:55:58,880 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/62fbc64c377241489a663af01ca2529d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d 2023-04-22 14:55:58,883 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 14:55:58,884 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/e3bd0e45a5444ced9f80cc8539a833aa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa 2023-04-22 14:55:58,887 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa, entries=150, sequenceid=644, filesize=11.9 K 2023-04-22 14:55:58,888 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 95b9d53f66a350cbea7732aecb64c699 in 854ms, sequenceid=644, compaction requested=true 2023-04-22 14:55:58,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:58,888 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:58,888 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:58,890 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,890 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:58,891 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,891 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:58,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:58,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@af3dc80[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:58,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:58,892 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,896 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,896 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:58,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:58,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:58,897 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:58,901 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:58,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:58,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:58,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:58,901 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:58,903 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,903 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:58,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,903 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:58,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:58,903 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,906 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:58,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:58,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:58,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:58,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:58,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:58,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:58,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:58,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:58,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:58,912 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:58,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:58,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:58,912 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:58,912 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:59,012 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:59,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:59,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:59,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:59,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:59,015 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:59,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:59,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:59,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,018 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,018 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:59,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:59,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,020 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:59,020 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:59,021 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:59,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175359245, max timestamp=1682175359269Num uniques 859; 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175359245, max timestamp=1682175359269Num uniques 859; 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175359245, max timestamp=1682175359269Num uniques 859; 2023-04-22 14:55:59,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:55:59,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175359292, max timestamp=1682175359324Num uniques 1068; 2023-04-22 14:55:59,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:55:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175359292, max timestamp=1682175359324Num uniques 1068; 2023-04-22 14:55:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175359292, max timestamp=1682175359324Num uniques 1068; 2023-04-22 14:55:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:55:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:55:59,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175419367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175419367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175419375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,430 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/e53bbacfff6249ff90f09ba11485e0a7 2023-04-22 14:55:59,449 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/e65ac5a512a743d1b6b24c9b244fd26a 2023-04-22 14:55:59,459 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/48b5aad7d6f34f48a3207ba1becb5d50 2023-04-22 14:55:59,463 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/e53bbacfff6249ff90f09ba11485e0a7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7 2023-04-22 14:55:59,467 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7, entries=100, sequenceid=654, filesize=9.5 K 2023-04-22 14:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/e65ac5a512a743d1b6b24c9b244fd26a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a 2023-04-22 14:55:59,471 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a, entries=100, sequenceid=654, filesize=9.5 K 2023-04-22 14:55:59,472 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/48b5aad7d6f34f48a3207ba1becb5d50 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50 2023-04-22 14:55:59,475 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50, entries=100, sequenceid=654, filesize=9.5 K 2023-04-22 14:55:59,476 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 456ms, sequenceid=654, compaction requested=true 2023-04-22 14:55:59,476 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:59,476 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,478 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195fa5c7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:59,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:59,480 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,481 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:59,481 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,481 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:59,482 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:59,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:59,485 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:59,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:59,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:59,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:59,485 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:59,488 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,488 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,488 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=140.89 KB heapSize=404.77 KB 2023-04-22 14:55:59,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:59,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:59,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:59,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:59,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:59,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:55:59,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,490 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,496 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:59,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:59,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:59,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/b94cf71d2cd7471583c1ddd59251c563 2023-04-22 14:55:59,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,500 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:59,500 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:59,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175419503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175419505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175419506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,600 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:59,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:59,603 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:59,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:59,603 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:59,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,604 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,605 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,605 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:59,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:59,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:59,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:59,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:59,609 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,610 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,610 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:59,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b25eb6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:59,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:59,611 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,613 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:59,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:59,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,614 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:59,614 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:59,615 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:59,618 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:59,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:59,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:59,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:59,618 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:59,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:59,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,628 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:59,628 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:59,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175419616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175419618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175419619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,728 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:59,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:59,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:59,732 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:59,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:59,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:59,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,733 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,734 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,734 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,734 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:59,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:59,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:59,735 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:59,735 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,735 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:59,735 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:59,735 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:59,736 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:59,736 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:59,736 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:59,736 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:59,737 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,737 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b93b9e9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:59,739 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:59,739 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,740 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:59,743 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:59,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:59,746 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:59,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:59,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:59,746 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:59,746 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:59,748 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,748 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,748 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,749 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,750 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,752 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,752 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:59,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,753 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:59,753 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,753 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:55:59,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175419829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175419830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:55:59,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175419830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,853 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:59,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:59,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:55:59,857 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:55:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:55:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:55:59,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,858 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,859 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:55:59,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:55:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:55:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:55:59,864 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f107bc7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:59,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:55:59,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,869 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:55:59,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,870 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:59,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:55:59,881 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,881 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,882 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:55:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:55:59,887 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:55:59,887 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:59,889 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,890 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:59,890 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:59,891 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:55:59,896 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:59,896 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:55:59,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:59,897 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:55:59,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:59,897 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:59,897 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:59,897 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:59,897 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:55:59,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:59,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/64bd52b6d9d843e7ae13994395cf3d12 2023-04-22 14:55:59,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/acc2c0368416449ba9515465f0d19b7c 2023-04-22 14:55:59,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/b94cf71d2cd7471583c1ddd59251c563 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563 2023-04-22 14:55:59,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563, entries=150, sequenceid=709, filesize=11.9 K 2023-04-22 14:55:59,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/64bd52b6d9d843e7ae13994395cf3d12 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12 2023-04-22 14:55:59,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12, entries=150, sequenceid=709, filesize=11.9 K 2023-04-22 14:55:59,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/acc2c0368416449ba9515465f0d19b7c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c 2023-04-22 14:55:59,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c, entries=150, sequenceid=709, filesize=11.9 K 2023-04-22 14:55:59,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~404.72 KB/414432, currentSize=40.25 KB/41220 for 95b9d53f66a350cbea7732aecb64c699 in 453ms, sequenceid=709, compaction requested=true 2023-04-22 14:55:59,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:59,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:59,942 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:59,942 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:59,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:59,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:59,944 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71762 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:59,944 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:55:59,944 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,944 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/0aaec5506fd24dff9a00f591e15890ac, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=70.1 K 2023-04-22 14:55:59,944 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71762 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:59,945 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:55:59,945 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0aaec5506fd24dff9a00f591e15890ac, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175354362 2023-04-22 14:55:59,945 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,945 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5aad955468884025847b23db50b1e94c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=70.1 K 2023-04-22 14:55:59,945 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 21ceea3f1d15494598f42e3eef11d8f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682175355590 2023-04-22 14:55:59,945 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 5aad955468884025847b23db50b1e94c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175354362 2023-04-22 14:55:59,945 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting b1201f4b73bc467a997e486d27547ff7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1682175356237 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7a213519f7774e24a262b5b49202038a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682175355590 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting eee65b65976347b2980e1b90e28d7829, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682175357412 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9270cf34a9274ab781bb799de398e70e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1682175356237 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e53bbacfff6249ff90f09ba11485e0a7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1682175358061 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 62fbc64c377241489a663af01ca2529d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682175357412 2023-04-22 14:55:59,946 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting b94cf71d2cd7471583c1ddd59251c563, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e65ac5a512a743d1b6b24c9b244fd26a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1682175358061 2023-04-22 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 64bd52b6d9d843e7ae13994395cf3d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:55:59,956 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#488 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:59,959 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#489 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:55:59,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:55:59,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:55:59,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:55:59,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:55:59,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:55:59,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/fcfd12b2ef0248819e863f6215a3874d 2023-04-22 14:55:59,978 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/0ab86115c66442ccbcfc7a3c74d104ec as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0ab86115c66442ccbcfc7a3c74d104ec 2023-04-22 14:55:59,981 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into 0ab86115c66442ccbcfc7a3c74d104ec(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:59,982 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:55:59,982 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=10, startTime=1682175359942; duration=0sec 2023-04-22 14:55:59,982 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:59,982 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-22 14:55:59,983 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71762 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-22 14:55:59,983 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:55:59,983 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:55:59,983 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/496b937aa3b54bc69999ff97f56f45c5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=70.1 K 2023-04-22 14:55:59,984 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 496b937aa3b54bc69999ff97f56f45c5, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1682175354362 2023-04-22 14:55:59,984 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 171dba8bf3594895a27a969367507f8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682175355590 2023-04-22 14:55:59,984 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 32f7820f553f4928b1ff5904368d6a1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1682175356237 2023-04-22 14:55:59,985 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e3bd0e45a5444ced9f80cc8539a833aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682175357412 2023-04-22 14:55:59,985 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 48b5aad7d6f34f48a3207ba1becb5d50, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1682175358061 2023-04-22 14:55:59,985 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting acc2c0368416449ba9515465f0d19b7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:55:59,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/16958de530dc41ed97dea1dfb9eebde9 2023-04-22 14:55:59,996 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#492 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:00,002 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,012 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,012 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,012 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,012 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,013 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,013 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,015 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,017 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/c51c1c7d9d7840e18b86c7fa48397e0c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/c51c1c7d9d7840e18b86c7fa48397e0c 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,017 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,017 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,017 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,017 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,019 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,019 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,021 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into c51c1c7d9d7840e18b86c7fa48397e0c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:00,021 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,022 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=10, startTime=1682175359942; duration=0sec 2023-04-22 14:56:00,022 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b058c7f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,024 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,037 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,037 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,037 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,046 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,046 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,046 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:00,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175360028, max timestamp=1682175360061Num uniques 859; 2023-04-22 14:56:00,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:00,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175360028, max timestamp=1682175360061Num uniques 859; 2023-04-22 14:56:00,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:00,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175360028, max timestamp=1682175360061Num uniques 859; 2023-04-22 14:56:00,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:00,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175420133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175420134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175420136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175420139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175420141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,146 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,149 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,153 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,156 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b13247[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,162 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,162 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,163 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,166 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,166 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,169 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,169 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,169 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,170 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,175 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,176 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,176 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175420242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175420247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175420247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,276 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,280 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,281 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,281 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,285 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3b9649[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,290 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,290 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,297 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,297 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,299 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,302 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,302 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,302 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,304 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,304 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,304 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,305 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,306 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,308 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,309 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,308 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,309 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,309 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,309 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,309 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,366 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/365f80c0cbe04e7aa1fd3ea051f94bab as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/365f80c0cbe04e7aa1fd3ea051f94bab 2023-04-22 14:56:00,371 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 365f80c0cbe04e7aa1fd3ea051f94bab(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:00,371 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:00,371 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=10, startTime=1682175359942; duration=0sec 2023-04-22 14:56:00,371 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:00,409 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,412 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,415 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,415 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,415 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,415 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/ed40d6e99fd3420abd0707349a85b852 2023-04-22 14:56:00,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a9080a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,421 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,421 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,422 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,423 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,423 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,427 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,427 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,429 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,429 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,429 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/fcfd12b2ef0248819e863f6215a3874d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d 2023-04-22 14:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,432 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d, entries=200, sequenceid=719, filesize=14.3 K 2023-04-22 14:56:00,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/16958de530dc41ed97dea1dfb9eebde9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,438 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,439 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,439 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:00,439 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,439 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9, entries=150, sequenceid=719, filesize=11.9 K 2023-04-22 14:56:00,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/ed40d6e99fd3420abd0707349a85b852 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852 2023-04-22 14:56:00,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852, entries=150, sequenceid=719, filesize=11.9 K 2023-04-22 14:56:00,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 95b9d53f66a350cbea7732aecb64c699 in 490ms, sequenceid=719, compaction requested=false 2023-04-22 14:56:00,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:00,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175360123, max timestamp=1682175360460Num uniques 1018; 2023-04-22 14:56:00,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175360123, max timestamp=1682175360460Num uniques 1018; 2023-04-22 14:56:00,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:56:00,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:56:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:00,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=64.85 KB heapSize=178.80 KB 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:56:00,462 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:00,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/00c67bbbb6e44d0e9a8032a88b52d4d7 2023-04-22 14:56:00,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/cb7802511b894b6bb8eedaf929c0ce18 2023-04-22 14:56:00,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175420511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175420513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175420516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,539 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,542 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,543 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,543 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,545 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,545 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,547 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,548 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,548 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,548 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,548 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,551 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55580254[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,553 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,553 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,554 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,554 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,554 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,556 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,562 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,562 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,562 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,562 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,562 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,563 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,563 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,563 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,564 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,565 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,568 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,569 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,569 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,569 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175420620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175420621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175420624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175420643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175420643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,669 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,673 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,673 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,675 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,675 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,675 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,675 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,676 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,676 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,676 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,676 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,676 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,677 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,677 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,677 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,677 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,678 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ba21b2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,680 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,682 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,682 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,682 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,685 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,687 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,687 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,687 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,687 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,687 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,689 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,689 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,689 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,690 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,691 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,693 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,694 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,694 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,694 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,794 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,797 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,797 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,798 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,799 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,799 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,801 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,801 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,801 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,801 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,801 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,808 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567164b2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,810 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,811 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,812 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,812 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,816 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,816 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,821 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,824 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,825 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175420830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175420830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:00,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175420832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,925 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:00,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:00,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/433fba31102e4c2ba88f171683460e8c 2023-04-22 14:56:00,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:00,928 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:00,928 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,928 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:00,928 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:00,929 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,929 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,931 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,932 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,932 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:00,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/00c67bbbb6e44d0e9a8032a88b52d4d7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7 2023-04-22 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:00,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:00,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:00,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:00,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:00,936 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:00,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7, entries=200, sequenceid=760, filesize=14.3 K 2023-04-22 14:56:00,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/cb7802511b894b6bb8eedaf929c0ce18 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18 2023-04-22 14:56:00,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,940 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c12cf2e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:00,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:00,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18, entries=150, sequenceid=760, filesize=11.9 K 2023-04-22 14:56:00,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/433fba31102e4c2ba88f171683460e8c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c 2023-04-22 14:56:00,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:00,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:00,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,946 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:00,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:00,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:00,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c, entries=150, sequenceid=760, filesize=11.9 K 2023-04-22 14:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~64.85 KB/66410, heapSize ~178.75 KB/183040, currentSize=114.05 KB/116790 for 95b9d53f66a350cbea7732aecb64c699 in 487ms, sequenceid=760, compaction requested=true 2023-04-22 14:56:00,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:00,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:00,949 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:00,949 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:00,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:00,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:00,950 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42732 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,950 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/A is initiating minor compaction (all files) 2023-04-22 14:56:00,950 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/A in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,950 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/365f80c0cbe04e7aa1fd3ea051f94bab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=41.7 K 2023-04-22 14:56:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,950 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:00,950 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/B is initiating minor compaction (all files) 2023-04-22 14:56:00,951 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/B in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,951 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0ab86115c66442ccbcfc7a3c74d104ec, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=37.0 K 2023-04-22 14:56:00,951 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 365f80c0cbe04e7aa1fd3ea051f94bab, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:56:00,951 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fcfd12b2ef0248819e863f6215a3874d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682175359501 2023-04-22 14:56:00,951 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0ab86115c66442ccbcfc7a3c74d104ec, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,952 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 00c67bbbb6e44d0e9a8032a88b52d4d7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682175360123 2023-04-22 14:56:00,952 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 16958de530dc41ed97dea1dfb9eebde9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682175359501 2023-04-22 14:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,952 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting cb7802511b894b6bb8eedaf929c0ce18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682175360123 2023-04-22 14:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:00,954 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:00,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:00,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:00,954 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:00,955 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:00,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:00,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,959 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:00,960 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,965 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#B#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:00,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:00,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:00,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:00,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:00,968 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:00,968 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:00,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:00,984 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/a4ecb2f2a49644f78c913bf8f30dd89c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a4ecb2f2a49644f78c913bf8f30dd89c 2023-04-22 14:56:00,988 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/B of 95b9d53f66a350cbea7732aecb64c699 into a4ecb2f2a49644f78c913bf8f30dd89c(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:00,988 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:00,988 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/B, priority=13, startTime=1682175360949; duration=0sec 2023-04-22 14:56:00,988 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:00,988 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:00,988 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:00,989 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 95b9d53f66a350cbea7732aecb64c699/C is initiating minor compaction (all files) 2023-04-22 14:56:00,989 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 95b9d53f66a350cbea7732aecb64c699/C in TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:00,989 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/c51c1c7d9d7840e18b86c7fa48397e0c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp, totalSize=37.0 K 2023-04-22 14:56:00,989 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c51c1c7d9d7840e18b86c7fa48397e0c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682175359364 2023-04-22 14:56:00,989 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting ed40d6e99fd3420abd0707349a85b852, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682175359501 2023-04-22 14:56:00,989 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 433fba31102e4c2ba88f171683460e8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1682175360123 2023-04-22 14:56:00,994 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 95b9d53f66a350cbea7732aecb64c699#C#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:01,068 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:01,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:01,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:01,072 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:01,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:01,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:01,072 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:01,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:01,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:01,074 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,074 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:01,074 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,074 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:01,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:01,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:01,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:01,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. started... 2023-04-22 14:56:01,076 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:56:01,077 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:01,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:01,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. as already flushing 2023-04-22 14:56:01,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175421152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175421152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175421159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175421260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175421260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175421269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,314 DEBUG [Thread-2880] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:51853 2023-04-22 14:56:01,314 DEBUG [Thread-2880] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:01,316 DEBUG [Thread-2888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x653bb8fc to 127.0.0.1:51853 2023-04-22 14:56:01,316 DEBUG [Thread-2888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:01,318 DEBUG [Thread-2886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:51853 2023-04-22 14:56:01,318 DEBUG [Thread-2886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:01,320 DEBUG [Thread-2884] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:51853 2023-04-22 14:56:01,320 DEBUG [Thread-2884] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:01,321 DEBUG [Thread-2882] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:51853 2023-04-22 14:56:01,321 DEBUG [Thread-2882] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:01,371 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/25e2bb09c18941469afdca3693dde0f6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/25e2bb09c18941469afdca3693dde0f6 2023-04-22 14:56:01,374 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/A of 95b9d53f66a350cbea7732aecb64c699 into 25e2bb09c18941469afdca3693dde0f6(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:01,374 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:01,374 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/A, priority=13, startTime=1682175360949; duration=0sec 2023-04-22 14:56:01,375 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:01,406 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/bf18590f832e417bb83270eee3742aad as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bf18590f832e417bb83270eee3742aad 2023-04-22 14:56:01,409 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 95b9d53f66a350cbea7732aecb64c699/C of 95b9d53f66a350cbea7732aecb64c699 into bf18590f832e417bb83270eee3742aad(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:01,409 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:01,409 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699., storeName=95b9d53f66a350cbea7732aecb64c699/C, priority=13, startTime=1682175360950; duration=0sec 2023-04-22 14:56:01,410 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:01,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175421466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175421466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175421472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,488 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/74e4e73061434a62ab416b23402fc25a 2023-04-22 14:56:01,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1682175421660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46840 deadline: 1682175421661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175421767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175421768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175421775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:01,897 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/6392be931169468eb6f9642e6f01e0c8 2023-04-22 14:56:02,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:02,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46830 deadline: 1682175422269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:02,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46788 deadline: 1682175422271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:02,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46804 deadline: 1682175422279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,306 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f10bfac4dcfb4fa581f143bf565ca1fa 2023-04-22 14:56:02,309 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/74e4e73061434a62ab416b23402fc25a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74e4e73061434a62ab416b23402fc25a 2023-04-22 14:56:02,312 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74e4e73061434a62ab416b23402fc25a, entries=150, sequenceid=781, filesize=11.9 K 2023-04-22 14:56:02,313 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/6392be931169468eb6f9642e6f01e0c8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6392be931169468eb6f9642e6f01e0c8 2023-04-22 14:56:02,316 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6392be931169468eb6f9642e6f01e0c8, entries=150, sequenceid=781, filesize=11.9 K 2023-04-22 14:56:02,317 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/f10bfac4dcfb4fa581f143bf565ca1fa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f10bfac4dcfb4fa581f143bf565ca1fa 2023-04-22 14:56:02,319 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f10bfac4dcfb4fa581f143bf565ca1fa, entries=150, sequenceid=781, filesize=11.9 K 2023-04-22 14:56:02,320 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 95b9d53f66a350cbea7732aecb64c699 in 1244ms, sequenceid=781, compaction requested=false 2023-04-22 14:56:02,320 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:02,320 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:02,322 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:02,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,322 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cd2c5a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:02,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:02,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:02,325 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,325 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:02,326 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:02,328 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:02,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:02,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:02,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:02,328 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:02,330 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,330 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:02,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,330 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:02,331 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:02,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:02,331 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,333 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,333 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:02,333 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:02,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:02,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:02,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:02,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:02,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:02,334 DEBUG [Thread-2879] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,334 DEBUG [Thread-2879] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:02,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:02,334 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:02,434 DEBUG [Thread-2879] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:03,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:03,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:56:03,273 DEBUG [Thread-2871] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:51853 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:03,273 DEBUG [Thread-2871] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:56:03,273 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:03,282 DEBUG [Thread-2869] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:51853 2023-04-22 14:56:03,282 DEBUG [Thread-2875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:51853 2023-04-22 14:56:03,283 DEBUG [Thread-2869] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,283 DEBUG [Thread-2875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,671 DEBUG [Thread-2873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:51853 2023-04-22 14:56:03,672 DEBUG [Thread-2873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,679 DEBUG [Thread-2877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:51853 2023-04-22 14:56:03,680 DEBUG [Thread-2877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 242 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 235 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:56:03,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=796 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/7f5e0413fc364011b343846d2777c937 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1623 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 4869 rows 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1633 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 4899 rows 2023-04-22 14:56:03,680 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1623 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 4869 rows 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1628 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 4884 rows 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(402): scanned 1621 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(403): verified 4860 rows 2023-04-22 14:56:03,681 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:56:03,681 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:51853 2023-04-22 14:56:03,681 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:03,682 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:56:03,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:56:03,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:03,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:03,687 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175363687"}]},"ts":"1682175363687"} 2023-04-22 14:56:03,688 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:56:03,690 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:56:03,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, UNASSIGN}] 2023-04-22 14:56:03,692 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, UNASSIGN 2023-04-22 14:56:03,693 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=95b9d53f66a350cbea7732aecb64c699, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:03,693 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175363693"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175363693"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175363693"}]},"ts":"1682175363693"} 2023-04-22 14:56:03,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:56:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:03,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:03,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 95b9d53f66a350cbea7732aecb64c699, disabling compactions & flushes 2023-04-22 14:56:03,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:03,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:04,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=796 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/ec20fbde95ca4e7d8cae936a73728ab8 2023-04-22 14:56:04,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:04,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=796 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/91bb4b54fae0458eab877d413c271353 2023-04-22 14:56:04,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/7f5e0413fc364011b343846d2777c937 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7f5e0413fc364011b343846d2777c937 2023-04-22 14:56:04,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7f5e0413fc364011b343846d2777c937, entries=150, sequenceid=796, filesize=11.9 K 2023-04-22 14:56:04,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/ec20fbde95ca4e7d8cae936a73728ab8 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ec20fbde95ca4e7d8cae936a73728ab8 2023-04-22 14:56:04,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ec20fbde95ca4e7d8cae936a73728ab8, entries=150, sequenceid=796, filesize=11.9 K 2023-04-22 14:56:04,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/91bb4b54fae0458eab877d413c271353 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/91bb4b54fae0458eab877d413c271353 2023-04-22 14:56:04,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/91bb4b54fae0458eab877d413c271353, entries=150, sequenceid=796, filesize=11.9 K 2023-04-22 14:56:04,518 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for 95b9d53f66a350cbea7732aecb64c699 in 1245ms, sequenceid=796, compaction requested=true 2023-04-22 14:56:04,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:04,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:04,519 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:04,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. after waiting 0 ms 2023-04-22 14:56:04,519 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:04,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 95b9d53f66a350cbea7732aecb64c699 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-22 14:56:04,519 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. because compaction request was cancelled 2023-04-22 14:56:04,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=A 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=B 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 95b9d53f66a350cbea7732aecb64c699, store=C 2023-04-22 14:56:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:04,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/27ad55bc66c54cc5936c1272c377a3c0 2023-04-22 14:56:04,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/8f64e4f8dbb34e30b7a8d38bd6c02938 2023-04-22 14:56:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:04,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=803 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/234da51e485f42bd89f883ab19f51206 2023-04-22 14:56:04,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/A/27ad55bc66c54cc5936c1272c377a3c0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/27ad55bc66c54cc5936c1272c377a3c0 2023-04-22 14:56:04,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/27ad55bc66c54cc5936c1272c377a3c0, entries=100, sequenceid=803, filesize=9.5 K 2023-04-22 14:56:04,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/B/8f64e4f8dbb34e30b7a8d38bd6c02938 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8f64e4f8dbb34e30b7a8d38bd6c02938 2023-04-22 14:56:04,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8f64e4f8dbb34e30b7a8d38bd6c02938, entries=100, sequenceid=803, filesize=9.5 K 2023-04-22 14:56:04,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/.tmp/C/234da51e485f42bd89f883ab19f51206 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/234da51e485f42bd89f883ab19f51206 2023-04-22 14:56:04,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/234da51e485f42bd89f883ab19f51206, entries=100, sequenceid=803, filesize=9.5 K 2023-04-22 14:56:04,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 95b9d53f66a350cbea7732aecb64c699 in 441ms, sequenceid=803, compaction requested=true 2023-04-22 14:56:04,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74db48047fd6494dacc3fd99211dd137, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d47c98d809e24bcd9306ab6ebc2e9a3e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/676b2cbb997a4aff8d642c92b810d771, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e3f2c42957164dc48859d13533d31297, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/bb07cd9a6ba6402c8383a9988c3f859f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/c1f771e2c0ca4bf08b0c7f0f88791cb5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/87b265d497464e46b12b3b8399af5a50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/0aaec5506fd24dff9a00f591e15890ac, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/365f80c0cbe04e7aa1fd3ea051f94bab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7] to archive 2023-04-22 14:56:04,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:04,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/f9f3b5a1eab54447a35453589a30f972 2023-04-22 14:56:04,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e6081b82374d4b319c7c8952198b4197 2023-04-22 14:56:04,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c221bce741a4503a57fa63137dd015a 2023-04-22 14:56:04,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/9d03938c223d4418b0ec15b332638bba 2023-04-22 14:56:04,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/570f33d025d14ff99496f8ef1ba3a0d5 2023-04-22 14:56:04,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74db48047fd6494dacc3fd99211dd137 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74db48047fd6494dacc3fd99211dd137 2023-04-22 14:56:04,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5e5c924e58f24da89ceb13d5ac76f909 2023-04-22 14:56:04,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ca56f48f6278465eb7ab06cef33664a2 2023-04-22 14:56:04,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/3aa5c94675d945618ce616c5d4fa5203 2023-04-22 14:56:04,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/aada44b0f7d645bcbcd093e2536626a9 2023-04-22 14:56:04,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/6c578f429b034a9eae681d35644b84d9 2023-04-22 14:56:04,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d47c98d809e24bcd9306ab6ebc2e9a3e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d47c98d809e24bcd9306ab6ebc2e9a3e 2023-04-22 14:56:04,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/09aeab316cb94e7eb6365de96716a067 2023-04-22 14:56:05,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d9f13581cefa4a938e3413ce6356441c 2023-04-22 14:56:05,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/4de492067a6141c2af8d63c9fff2bd3f 2023-04-22 14:56:05,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/676b2cbb997a4aff8d642c92b810d771 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/676b2cbb997a4aff8d642c92b810d771 2023-04-22 14:56:05,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/23bafacd1a124823acedd59c56742721 2023-04-22 14:56:05,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/8b590197a05a44f1a6451e4d236ecaf7 2023-04-22 14:56:05,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/34b905f0c0cc445e84b8906482ec2ebe 2023-04-22 14:56:05,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e3f2c42957164dc48859d13533d31297 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e3f2c42957164dc48859d13533d31297 2023-04-22 14:56:05,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/be3b6f1954f64fe1a040f81541fd36d1 2023-04-22 14:56:05,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/5191926c54de4c39ba2fe74003392781 2023-04-22 14:56:05,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/bb07cd9a6ba6402c8383a9988c3f859f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/bb07cd9a6ba6402c8383a9988c3f859f 2023-04-22 14:56:05,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fe2040a17e144e05adfa6ebc15b5d1a8 2023-04-22 14:56:05,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/d19487347d3a4b968bd9bd600def3fdd 2023-04-22 14:56:05,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7494657b0b53430a8dae5a79995d59cd 2023-04-22 14:56:05,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/c1f771e2c0ca4bf08b0c7f0f88791cb5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/c1f771e2c0ca4bf08b0c7f0f88791cb5 2023-04-22 14:56:05,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/ae4ee425efba48bcb9c1df130e67b6ee 2023-04-22 14:56:05,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/81e6784eb36d4faab34bec86e1b3773e 2023-04-22 14:56:05,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/87b265d497464e46b12b3b8399af5a50 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/87b265d497464e46b12b3b8399af5a50 2023-04-22 14:56:05,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/43b7d615f9584234803299212166ad1b 2023-04-22 14:56:05,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/0aaec5506fd24dff9a00f591e15890ac to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/0aaec5506fd24dff9a00f591e15890ac 2023-04-22 14:56:05,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/a7c49addf60e448aae151502bad56c04 2023-04-22 14:56:05,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/21ceea3f1d15494598f42e3eef11d8f2 2023-04-22 14:56:05,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b1201f4b73bc467a997e486d27547ff7 2023-04-22 14:56:05,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/eee65b65976347b2980e1b90e28d7829 2023-04-22 14:56:05,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/e53bbacfff6249ff90f09ba11485e0a7 2023-04-22 14:56:05,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/365f80c0cbe04e7aa1fd3ea051f94bab to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/365f80c0cbe04e7aa1fd3ea051f94bab 2023-04-22 14:56:05,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/b94cf71d2cd7471583c1ddd59251c563 2023-04-22 14:56:05,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/fcfd12b2ef0248819e863f6215a3874d 2023-04-22 14:56:05,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/00c67bbbb6e44d0e9a8032a88b52d4d7 2023-04-22 14:56:05,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/be89f40483b842b4a9d90dfeddacc662, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0bf689dcabcf4989ac4547e2740bca85, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6246327f3193468ba8c5646031996ed4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/61edda45db5c47e3bd62ead6498ebe06, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b8f96ead33154147a732e59a872e31d8, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a91c2afed9834317b39cb0e30105d947, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/1c26375dab0248d09913debf45c72ddc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5aad955468884025847b23db50b1e94c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0ab86115c66442ccbcfc7a3c74d104ec, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18] to archive 2023-04-22 14:56:05,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:05,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/48e22656d7524ea382ff988704f8974c 2023-04-22 14:56:05,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8de943aa4a064bb7bc9fdc4236726adf 2023-04-22 14:56:05,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/c66e3a7da06d401f8526fef0cbcc1226 2023-04-22 14:56:05,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2d1ba3039698444d9b90e3c4e20eadd3 2023-04-22 14:56:05,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/be89f40483b842b4a9d90dfeddacc662 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/be89f40483b842b4a9d90dfeddacc662 2023-04-22 14:56:05,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ecf0989249c04e92b6096d7f864d060d 2023-04-22 14:56:05,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cc93b097ca6a4d62a81d1d929d965ee3 2023-04-22 14:56:05,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/57569e2449a0437696fc915db5a2f786 2023-04-22 14:56:05,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5df8c2c59c5d4f94bc25091e19db17eb 2023-04-22 14:56:05,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7bbd023cf1ce4cf19dac8ac6f3424eb6 2023-04-22 14:56:05,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0bf689dcabcf4989ac4547e2740bca85 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0bf689dcabcf4989ac4547e2740bca85 2023-04-22 14:56:05,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6ec7ac5c807a4042a6b7dac7cb5211a7 2023-04-22 14:56:05,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/13f67f1085e242788f6308f70234ec3f 2023-04-22 14:56:05,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d4fe610657c24677b42b8afa97615692 2023-04-22 14:56:05,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6246327f3193468ba8c5646031996ed4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6246327f3193468ba8c5646031996ed4 2023-04-22 14:56:05,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5d1d79d7037140508121b6aeaff09341 2023-04-22 14:56:05,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b45d81dfa04b49199e97ea32c18bb667 2023-04-22 14:56:05,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/849bef887329497699dafbc44d317fba 2023-04-22 14:56:05,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/61edda45db5c47e3bd62ead6498ebe06 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/61edda45db5c47e3bd62ead6498ebe06 2023-04-22 14:56:05,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/bb120cb1b4704fe2b4de0921bea40d16 2023-04-22 14:56:05,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/31079928ae6d4861a1209a0b70148f1c 2023-04-22 14:56:05,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b8f96ead33154147a732e59a872e31d8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/b8f96ead33154147a732e59a872e31d8 2023-04-22 14:56:05,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d919266a2f3a4e429f3a832560353721 2023-04-22 14:56:05,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3a29b6865c2c4a15868d8723e902026c 2023-04-22 14:56:05,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/d6477fc33db24fdb949a0700b930472e 2023-04-22 14:56:05,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a91c2afed9834317b39cb0e30105d947 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a91c2afed9834317b39cb0e30105d947 2023-04-22 14:56:05,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/58caa6a0e31c4ee0aacb2d5938b93841 2023-04-22 14:56:05,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/74afd1a2bcd44a8a9364a77539ab9e17 2023-04-22 14:56:05,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/1c26375dab0248d09913debf45c72ddc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/1c26375dab0248d09913debf45c72ddc 2023-04-22 14:56:05,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7f8e5741459e45b0b0ffa423c9457103 2023-04-22 14:56:05,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/2e714c6fa6de4293b107ed6ab9bcdbee 2023-04-22 14:56:05,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5aad955468884025847b23db50b1e94c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/5aad955468884025847b23db50b1e94c 2023-04-22 14:56:05,059 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/3be868c385364a2799d350c7feb74d25 2023-04-22 14:56:05,060 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/7a213519f7774e24a262b5b49202038a 2023-04-22 14:56:05,063 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/9270cf34a9274ab781bb799de398e70e 2023-04-22 14:56:05,064 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/62fbc64c377241489a663af01ca2529d 2023-04-22 14:56:05,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/e65ac5a512a743d1b6b24c9b244fd26a 2023-04-22 14:56:05,066 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0ab86115c66442ccbcfc7a3c74d104ec to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/0ab86115c66442ccbcfc7a3c74d104ec 2023-04-22 14:56:05,067 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/64bd52b6d9d843e7ae13994395cf3d12 2023-04-22 14:56:05,068 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/16958de530dc41ed97dea1dfb9eebde9 2023-04-22 14:56:05,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/cb7802511b894b6bb8eedaf929c0ce18 2023-04-22 14:56:05,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d07292a8351c4207805fc67f428ccf4d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dd2371d644db4497b346e1d011249a97, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/948997a843ed4e61945d8dd96a3fe8c5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d55e06338ad84835b5bec5bf435c473d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/47994eebb9624843861836cf2eb05c89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a3254f20e9134017b5c188271b2b638c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/079973d17e55423fbf68992a7f769b61, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/496b937aa3b54bc69999ff97f56f45c5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/c51c1c7d9d7840e18b86c7fa48397e0c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c] to archive 2023-04-22 14:56:05,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:05,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3f16341f9b654aed9a136600b96a6fb2 2023-04-22 14:56:05,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/56a0812e039c415fa7178e0e82376585 2023-04-22 14:56:05,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a424bd286a3f4fd9b513d9876371fc75 2023-04-22 14:56:05,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/cf97e6a7e6154fa28f5a746382b163ab 2023-04-22 14:56:05,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d07292a8351c4207805fc67f428ccf4d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d07292a8351c4207805fc67f428ccf4d 2023-04-22 14:56:05,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/1203a5804a204feeb1ab3f14367c7846 2023-04-22 14:56:05,080 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f15b1c8bc820453781d4cc5c6e3b7040 2023-04-22 14:56:05,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/910e472d92564acea0cd60f02667b538 2023-04-22 14:56:05,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/758d21245f63409ca9ba39986245c3da 2023-04-22 14:56:05,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/eb46564aefb544929eaaf35e216596c4 2023-04-22 14:56:05,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dd2371d644db4497b346e1d011249a97 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dd2371d644db4497b346e1d011249a97 2023-04-22 14:56:05,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/28cb3ea57db9472d860fd67a7f8904a5 2023-04-22 14:56:05,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3411fc90f3be4f2c81d9ca34dd365619 2023-04-22 14:56:05,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3a14762f1c474dd08d5652d99d9e945e 2023-04-22 14:56:05,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/948997a843ed4e61945d8dd96a3fe8c5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/948997a843ed4e61945d8dd96a3fe8c5 2023-04-22 14:56:05,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e2ec5a0884bf46f9ba3b9305ee222122 2023-04-22 14:56:05,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/6ae3c1c282824f33a458abba24d59178 2023-04-22 14:56:05,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bed6c41dba984c7ba2df7a8778bc518c 2023-04-22 14:56:05,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d55e06338ad84835b5bec5bf435c473d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d55e06338ad84835b5bec5bf435c473d 2023-04-22 14:56:05,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/3aacfd492ece4d57a693bdb66bef3359 2023-04-22 14:56:05,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/69828d7286b24c1bb3517bf9ad805452 2023-04-22 14:56:05,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/d89b4beaced84d768050e3832fd3bb91 2023-04-22 14:56:05,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/47994eebb9624843861836cf2eb05c89 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/47994eebb9624843861836cf2eb05c89 2023-04-22 14:56:05,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e5bac47f38d44561a75dd5f218bc3d33 2023-04-22 14:56:05,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/9e3f6180217b4c129a567827df4e0040 2023-04-22 14:56:05,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a3254f20e9134017b5c188271b2b638c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/a3254f20e9134017b5c188271b2b638c 2023-04-22 14:56:05,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e0b7efff8f974ffbae5ed344e877d322 2023-04-22 14:56:05,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/09c4b08ce2e246c4b17f28d052a7cfd9 2023-04-22 14:56:05,097 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/079973d17e55423fbf68992a7f769b61 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/079973d17e55423fbf68992a7f769b61 2023-04-22 14:56:05,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f2d5be944a96498388eb1ed347bba178 2023-04-22 14:56:05,099 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/dca67e990a6a4d3c930304a69d84b29c 2023-04-22 14:56:05,100 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/496b937aa3b54bc69999ff97f56f45c5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/496b937aa3b54bc69999ff97f56f45c5 2023-04-22 14:56:05,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f180baa212774621b9f7ef7a5ce1df60 2023-04-22 14:56:05,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/171dba8bf3594895a27a969367507f8c 2023-04-22 14:56:05,102 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/32f7820f553f4928b1ff5904368d6a1a 2023-04-22 14:56:05,103 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/e3bd0e45a5444ced9f80cc8539a833aa 2023-04-22 14:56:05,104 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/48b5aad7d6f34f48a3207ba1becb5d50 2023-04-22 14:56:05,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/c51c1c7d9d7840e18b86c7fa48397e0c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/c51c1c7d9d7840e18b86c7fa48397e0c 2023-04-22 14:56:05,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/acc2c0368416449ba9515465f0d19b7c 2023-04-22 14:56:05,106 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/ed40d6e99fd3420abd0707349a85b852 2023-04-22 14:56:05,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/433fba31102e4c2ba88f171683460e8c 2023-04-22 14:56:05,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/recovered.edits/806.seqid, newMaxSeqId=806, maxSeqId=1 2023-04-22 14:56:05,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699. 2023-04-22 14:56:05,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 95b9d53f66a350cbea7732aecb64c699: 2023-04-22 14:56:05,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:05,114 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=95b9d53f66a350cbea7732aecb64c699, regionState=CLOSED 2023-04-22 14:56:05,114 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175365114"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175365114"}]},"ts":"1682175365114"} 2023-04-22 14:56:05,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-22 14:56:05,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 95b9d53f66a350cbea7732aecb64c699, server=jenkins-hbase4.apache.org,32935,1682175218140 in 1.4210 sec 2023-04-22 14:56:05,117 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-22 14:56:05,117 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=95b9d53f66a350cbea7732aecb64c699, UNASSIGN in 1.4250 sec 2023-04-22 14:56:05,117 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175365117"}]},"ts":"1682175365117"} 2023-04-22 14:56:05,118 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:56:05,120 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:56:05,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4380 sec 2023-04-22 14:56:05,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-22 14:56:05,792 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-22 14:56:05,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:56:05,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,794 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,795 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-22 14:56:05,800 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:05,801 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/recovered.edits] 2023-04-22 14:56:05,803 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/25e2bb09c18941469afdca3693dde0f6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/25e2bb09c18941469afdca3693dde0f6 2023-04-22 14:56:05,805 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/27ad55bc66c54cc5936c1272c377a3c0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/27ad55bc66c54cc5936c1272c377a3c0 2023-04-22 14:56:05,806 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74e4e73061434a62ab416b23402fc25a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/74e4e73061434a62ab416b23402fc25a 2023-04-22 14:56:05,807 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7f5e0413fc364011b343846d2777c937 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/A/7f5e0413fc364011b343846d2777c937 2023-04-22 14:56:05,809 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6392be931169468eb6f9642e6f01e0c8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/6392be931169468eb6f9642e6f01e0c8 2023-04-22 14:56:05,810 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8f64e4f8dbb34e30b7a8d38bd6c02938 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/8f64e4f8dbb34e30b7a8d38bd6c02938 2023-04-22 14:56:05,811 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a4ecb2f2a49644f78c913bf8f30dd89c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/a4ecb2f2a49644f78c913bf8f30dd89c 2023-04-22 14:56:05,812 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ec20fbde95ca4e7d8cae936a73728ab8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/B/ec20fbde95ca4e7d8cae936a73728ab8 2023-04-22 14:56:05,815 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/234da51e485f42bd89f883ab19f51206 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/234da51e485f42bd89f883ab19f51206 2023-04-22 14:56:05,816 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/91bb4b54fae0458eab877d413c271353 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/91bb4b54fae0458eab877d413c271353 2023-04-22 14:56:05,817 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bf18590f832e417bb83270eee3742aad to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/bf18590f832e417bb83270eee3742aad 2023-04-22 14:56:05,818 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f10bfac4dcfb4fa581f143bf565ca1fa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/C/f10bfac4dcfb4fa581f143bf565ca1fa 2023-04-22 14:56:05,821 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/recovered.edits/806.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699/recovered.edits/806.seqid 2023-04-22 14:56:05,821 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/95b9d53f66a350cbea7732aecb64c699 2023-04-22 14:56:05,821 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:56:05,823 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,825 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:56:05,826 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:56:05,827 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,827 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:56:05,827 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175365827"}]},"ts":"9223372036854775807"} 2023-04-22 14:56:05,829 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:56:05,829 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 95b9d53f66a350cbea7732aecb64c699, NAME => 'TestAcidGuarantees,,1682175339083.95b9d53f66a350cbea7732aecb64c699.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:56:05,829 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:56:05,829 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175365829"}]},"ts":"9223372036854775807"} 2023-04-22 14:56:05,830 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:56:05,832 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,833 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-22 14:56:05,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-22 14:56:05,896 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-22 14:56:05,911 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=591 (was 550) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12705 (was 12289) - AvailableMemoryMB LEAK? - 2023-04-22 14:56:05,923 INFO [Listener at localhost/45141] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=591, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=170, AvailableMemoryMB=12704 2023-04-22 14:56:05,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:56:05,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:56:05,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:05,927 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:56:05,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-22 14:56:05,928 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:56:05,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:05,929 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:05,930 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb empty. 2023-04-22 14:56:05,930 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:05,930 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:56:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:06,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:06,337 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:56:06,338 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 89f2897412e14388617caefccca798eb, NAME => 'TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp 2023-04-22 14:56:06,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 89f2897412e14388617caefccca798eb, disabling compactions & flushes 2023-04-22 14:56:06,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. after waiting 0 ms 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:06,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:06,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:06,746 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:56:06,747 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175366747"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175366747"}]},"ts":"1682175366747"} 2023-04-22 14:56:06,748 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:56:06,748 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:56:06,749 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175366748"}]},"ts":"1682175366748"} 2023-04-22 14:56:06,749 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-22 14:56:06,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, ASSIGN}] 2023-04-22 14:56:06,754 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, ASSIGN 2023-04-22 14:56:06,754 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=false 2023-04-22 14:56:06,906 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:06,906 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175366906"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175366906"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175366906"}]},"ts":"1682175366906"} 2023-04-22 14:56:06,907 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:56:07,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:07,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:07,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 89f2897412e14388617caefccca798eb, NAME => 'TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:56:07,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:56:07,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,065 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,066 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A 2023-04-22 14:56:07,066 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A 2023-04-22 14:56:07,067 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:07,067 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName A 2023-04-22 14:56:07,067 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:07,067 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,068 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B 2023-04-22 14:56:07,069 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B 2023-04-22 14:56:07,069 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:07,069 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName B 2023-04-22 14:56:07,070 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:07,070 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,071 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C 2023-04-22 14:56:07,071 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C 2023-04-22 14:56:07,071 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:07,071 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName C 2023-04-22 14:56:07,072 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:07,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:07,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:56:07,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:07,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:56:07,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 89f2897412e14388617caefccca798eb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71471014, jitterRate=0.06500110030174255}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:56:07,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:07,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., pid=56, masterSystemTime=1682175367058 2023-04-22 14:56:07,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:07,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:07,080 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:07,080 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175367080"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175367080"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175367080"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175367080"}]},"ts":"1682175367080"} 2023-04-22 14:56:07,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-22 14:56:07,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 in 174 msec 2023-04-22 14:56:07,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-22 14:56:07,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, ASSIGN in 330 msec 2023-04-22 14:56:07,084 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:56:07,084 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175367084"}]},"ts":"1682175367084"} 2023-04-22 14:56:07,085 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-22 14:56:07,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:56:07,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-22 14:56:08,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-22 14:56:08,033 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-22 14:56:08,034 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,039 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,041 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:08,042 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44874, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:08,043 DEBUG [Listener at localhost/45141] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:56:08,045 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:56:08,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-22 14:56:08,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:56:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:08,461 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:56:08,462 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-22 14:56:08,462 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-22 14:56:08,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-22 14:56:08,466 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, REOPEN/MOVE}] 2023-04-22 14:56:08,467 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, REOPEN/MOVE 2023-04-22 14:56:08,468 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:08,468 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175368468"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175368468"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175368468"}]},"ts":"1682175368468"} 2023-04-22 14:56:08,469 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:56:08,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 89f2897412e14388617caefccca798eb, disabling compactions & flushes 2023-04-22 14:56:08,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. after waiting 0 ms 2023-04-22 14:56:08,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 14:56:08,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:08,625 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 89f2897412e14388617caefccca798eb to self. 2023-04-22 14:56:08,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,626 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=CLOSED 2023-04-22 14:56:08,626 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175368626"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175368626"}]},"ts":"1682175368626"} 2023-04-22 14:56:08,628 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-22 14:56:08,628 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 in 158 msec 2023-04-22 14:56:08,629 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,32935,1682175218140; forceNewPlan=false, retain=true 2023-04-22 14:56:08,779 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:08,779 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175368779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175368779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175368779"}]},"ts":"1682175368779"} 2023-04-22 14:56:08,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:56:08,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 89f2897412e14388617caefccca798eb, NAME => 'TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:56:08,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:56:08,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,936 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,937 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A 2023-04-22 14:56:08,937 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A 2023-04-22 14:56:08,937 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:08,938 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName A 2023-04-22 14:56:08,939 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:08,939 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,939 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B 2023-04-22 14:56:08,940 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B 2023-04-22 14:56:08,940 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:08,940 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName B 2023-04-22 14:56:08,940 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:08,940 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,941 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C 2023-04-22 14:56:08,941 DEBUG [StoreOpener-89f2897412e14388617caefccca798eb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C 2023-04-22 14:56:08,941 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-22 14:56:08,941 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89f2897412e14388617caefccca798eb columnFamilyName C 2023-04-22 14:56:08,942 INFO [StoreOpener-89f2897412e14388617caefccca798eb-1] regionserver.HStore(310): Store=89f2897412e14388617caefccca798eb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:56:08,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-22 14:56:08,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 89f2897412e14388617caefccca798eb 2023-04-22 14:56:08,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 89f2897412e14388617caefccca798eb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73691690, jitterRate=0.09809175133705139}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-22 14:56:08,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:08,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., pid=61, masterSystemTime=1682175368932 2023-04-22 14:56:08,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:08,948 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:08,948 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175368948"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175368948"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175368948"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175368948"}]},"ts":"1682175368948"} 2023-04-22 14:56:08,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-22 14:56:08,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 in 168 msec 2023-04-22 14:56:08,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-22 14:56:08,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, REOPEN/MOVE in 484 msec 2023-04-22 14:56:08,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-22 14:56:08,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-22 14:56:08,954 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-22 14:56:08,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-22 14:56:08,956 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,960 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,961 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,965 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,965 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,969 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,969 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,974 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,974 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,978 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,978 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,982 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,982 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,991 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,991 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:08,998 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:08,998 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:09,002 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:09,002 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:51853 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:56:09,006 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:56:09,014 DEBUG [hconnection-0x5fc58c1f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,015 DEBUG [hconnection-0xb2160ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,015 DEBUG [hconnection-0x2e6ffcc8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,020 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,021 DEBUG [hconnection-0x25b6662-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,021 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,022 DEBUG [hconnection-0x9433678-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,022 DEBUG [hconnection-0xb99cc6d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,022 DEBUG [hconnection-0x354b9c3f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,024 DEBUG [hconnection-0x5d9c7890-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,025 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,028 DEBUG [hconnection-0x6a607dbc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,028 DEBUG [hconnection-0x537e0eac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:56:09,031 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44908, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,032 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,032 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,032 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,032 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,033 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,040 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,040 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,041 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,041 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:56:09,041 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,045 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:09,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 14:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,047 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,047 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:09,048 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:09,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:09,050 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,050 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,051 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,051 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3f3360[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,054 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,054 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,056 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,060 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,060 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,060 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,062 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175429059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175429060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175369054, max timestamp=1682175369062Num uniques 859; 2023-04-22 14:56:09,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175369054, max timestamp=1682175369062Num uniques 859; 2023-04-22 14:56:09,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:09,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:09,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175369054, max timestamp=1682175369062Num uniques 859; 2023-04-22 14:56:09,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:09,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,065 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,068 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,069 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,069 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175429077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175429078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175429079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175429163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175429164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,169 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,172 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,173 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,174 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,174 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,176 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,176 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,176 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,176 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,176 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,177 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,178 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,179 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513682a1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,181 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,181 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,183 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175429182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,183 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,183 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175429182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,183 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,183 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,184 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175429182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,185 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,185 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,187 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,187 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,189 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,191 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,193 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,193 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,193 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,193 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,293 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,297 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,297 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,298 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,300 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,300 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,301 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,302 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,302 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,303 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,303 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,303 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,306 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,306 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,306 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,306 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b974c16[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,308 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,308 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,309 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,309 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,309 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,310 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,313 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,313 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,313 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,313 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,313 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,323 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,324 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,324 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,324 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,324 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175429367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175429368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175429384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175429384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175429385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,424 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,428 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,428 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,428 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,428 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,429 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,429 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,430 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,430 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,431 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,432 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,432 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,432 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,432 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,438 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b981cc2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,440 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,440 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,441 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,441 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,441 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,441 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,443 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,446 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,446 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,446 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,448 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,448 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,448 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,451 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,454 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,454 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,454 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,455 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,455 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,455 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,479 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422bc5389713e7549438e38e14b3fb17c46 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bc5389713e7549438e38e14b3fb17c46 2023-04-22 14:56:09,555 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,559 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,560 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,560 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,561 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,561 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,562 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,564 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,565 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e499ce2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,566 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,566 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,568 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,568 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,568 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,569 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,569 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,572 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,572 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,573 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,573 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,574 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,575 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,576 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,579 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,579 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,580 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,580 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,580 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175429670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175429672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,680 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,683 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,683 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,683 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,683 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,684 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,684 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,686 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,686 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,687 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,687 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,688 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,688 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,690 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,690 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175429689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175429689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:09,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175429690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0f2c00[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,693 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,694 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,695 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,695 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,696 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,699 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,699 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,699 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,702 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,702 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,703 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,703 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,705 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,707 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,707 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,708 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,708 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,808 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,811 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,811 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,812 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,812 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,814 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,815 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,818 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a59328b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,820 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,820 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,822 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,822 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,822 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,826 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,828 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,829 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,829 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,830 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,830 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,830 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,831 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,832 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,836 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,836 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,836 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:09,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,836 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,837 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,889 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/547361f3a5674052829477f012323880 2023-04-22 14:56:09,937 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:09,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:09,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:09,940 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:09,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:09,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:09,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,943 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:09,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:09,945 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,945 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:09,945 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:09,945 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:09,945 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:09,947 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715c410e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:09,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:09,949 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,950 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:09,952 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:09,952 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,952 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:09,952 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:09,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:09,953 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:09,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:09,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:09,953 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:09,953 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:09,955 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,955 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:09,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,955 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:09,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,956 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,959 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,961 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:09,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:09,962 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:09,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:09,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:09,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:09,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:09,962 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:09,962 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:09,962 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:10,062 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:10,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:10,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:10,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:10,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:10,066 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:10,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:10,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:10,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,066 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,067 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,067 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,068 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,068 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:10,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:10,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:10,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:10,069 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:10,070 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:10,071 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,072 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73706d4b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:10,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:10,073 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,075 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:10,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,077 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:10,082 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:10,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:10,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:10,082 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:10,082 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:10,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,084 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,085 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:10,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:10,087 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,088 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:10,088 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:10,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,088 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:10,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175430174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:10,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175430180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,188 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:10,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:10,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:10,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:10,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:10,191 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:10,191 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,191 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:10,192 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:10,192 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,192 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,194 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,194 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:10,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175430194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:10,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175430194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:10,196 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:10,197 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,198 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:10,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175430196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cd3009[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:10,201 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:10,201 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,202 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:10,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:10,203 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:10,206 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:10,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:10,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:10,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:10,206 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:10,211 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,211 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,213 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:10,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:10,216 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,216 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,216 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:10,216 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:10,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/0fd69fb02f8f41a2945f643cb6439cda 2023-04-22 14:56:10,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/2c84140fd88c40199fd7808b4a239b48 2023-04-22 14:56:10,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/547361f3a5674052829477f012323880 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880 2023-04-22 14:56:10,317 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:10,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:10,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880, entries=150, sequenceid=16, filesize=25.1 K 2023-04-22 14:56:10,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/0fd69fb02f8f41a2945f643cb6439cda as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda 2023-04-22 14:56:10,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:10,320 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:10,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,322 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,322 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda, entries=150, sequenceid=16, filesize=11.6 K 2023-04-22 14:56:10,322 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/2c84140fd88c40199fd7808b4a239b48 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48 2023-04-22 14:56:10,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:10,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:10,327 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c393d1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:10,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:10,329 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48, entries=150, sequenceid=16, filesize=11.6 K 2023-04-22 14:56:10,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 89f2897412e14388617caefccca798eb in 1284ms, sequenceid=16, compaction requested=false 2023-04-22 14:56:10,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:10,330 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:10,330 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,330 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:10,332 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:10,335 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:10,335 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:10,335 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:10,335 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:10,335 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:10,337 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,337 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,337 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,338 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,340 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,344 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,344 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,345 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:10,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:10,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,345 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:10,345 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,345 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:10,445 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:10,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:10,449 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:10,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:10,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:10,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,449 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,450 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,450 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,451 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,451 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:10,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:10,452 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:10,452 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:10,452 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:10,452 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:10,452 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:10,453 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:10,453 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:10,648 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:56:10,865 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422fabe015e5a88474eae5642db562ee6b5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fabe015e5a88474eae5642db562ee6b5 2023-04-22 14:56:10,873 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7623c3b769d94a0d8613ab5f35543c35 2023-04-22 14:56:10,888 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/a4b49550a4fe46a2bba0d79058a8af10 2023-04-22 14:56:10,900 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3a17af15104441c09cc16ccaa7c4135d 2023-04-22 14:56:10,906 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7623c3b769d94a0d8613ab5f35543c35 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35 2023-04-22 14:56:10,910 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35, entries=150, sequenceid=49, filesize=25.1 K 2023-04-22 14:56:10,911 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/a4b49550a4fe46a2bba0d79058a8af10 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10 2023-04-22 14:56:10,915 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 14:56:10,916 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3a17af15104441c09cc16ccaa7c4135d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d 2023-04-22 14:56:10,925 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d, entries=150, sequenceid=49, filesize=11.6 K 2023-04-22 14:56:10,926 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=0 B/0 for 89f2897412e14388617caefccca798eb in 473ms, sequenceid=49, compaction requested=false 2023-04-22 14:56:10,926 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:10,926 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:10,926 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:10,928 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,928 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:10,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7104eab6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:10,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:10,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,932 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:10,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:10,932 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:10,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,934 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:10,937 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:10,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:10,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:10,937 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:10,937 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:10,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,939 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:10,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:10,940 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,941 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,944 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:10,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:10,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:10,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:10,945 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:10,945 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:10,945 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:10,945 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:10,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:10,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:10,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,045 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,049 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,051 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,052 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:11,084 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:11,085 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:11,087 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,087 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,088 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ecbe8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:11,089 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:11,090 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,091 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,091 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:11,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,091 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:11,091 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:11,091 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:11,092 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,092 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,093 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:11,095 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:11,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:11,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:11,096 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:11,096 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:11,098 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,098 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,098 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,098 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,106 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:11,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:11,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:11,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,106 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:11,106 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:11,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,107 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,207 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,210 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,210 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,211 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,211 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,212 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,212 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,216 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:11,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:11,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682175371234, max timestamp=1682175371241Num uniques 859; 2023-04-22 14:56:11,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 14:56:11,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682175371234, max timestamp=1682175371241Num uniques 859; 2023-04-22 14:56:11,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 14:56:11,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682175371234, max timestamp=1682175371241Num uniques 859; 2023-04-22 14:56:11,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-22 14:56:11,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:11,244 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:11,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:11,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:11,244 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:11,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,246 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:11,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496bfeb4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:11,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:11,249 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,250 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:11,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:11,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,250 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:11,250 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:11,251 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:11,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:11,254 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:11,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:11,254 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:11,255 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:11,255 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:11,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,257 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,263 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,262 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:11,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:11,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,263 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:11,263 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:11,263 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175431257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175431258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175431261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,266 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042256809a7888774577a9f6c1833cd30c1d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256809a7888774577a9f6c1833cd30c1d 2023-04-22 14:56:11,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175431262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175431263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,363 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175431365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,367 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175431365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175431366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,368 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175431367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175431368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,370 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,370 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,373 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,373 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,374 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,374 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:11,374 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:11,375 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,375 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,376 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c8b153[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:11,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:11,378 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,380 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,380 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:11,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:11,381 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:11,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,381 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:11,382 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:11,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:11,385 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:11,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:11,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:11,385 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:11,385 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:11,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,387 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,388 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,389 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:11,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:11,391 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,392 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:11,392 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:11,392 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,392 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,492 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,496 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,497 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,499 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,500 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,501 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,501 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,501 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,501 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,501 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:11,502 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:11,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:11,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:11,502 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:11,503 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,504 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cfac46[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:11,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:11,506 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:11,507 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:11,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,507 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:11,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:11,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,509 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:11,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:11,511 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:11,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:11,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:11,512 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:11,512 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:11,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,514 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,515 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,516 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:11,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:11,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:11,519 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,519 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:11,519 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,519 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:11,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,520 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175431569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175431569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175431570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175431570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175431571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,620 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,623 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,628 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,629 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,629 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,629 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:11,630 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:11,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:11,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:11,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:11,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,631 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,632 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16890a1c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:11,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:11,634 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,635 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:11,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:11,635 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:11,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,635 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:11,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:11,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:11,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:11,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:11,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:11,640 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:11,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,643 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:11,644 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,648 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:11,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:11,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:11,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:11,648 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,649 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:11,649 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:11,649 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:11,649 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:11,674 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7121708895384e31b6f5feabe407ef80 2023-04-22 14:56:11,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/be8522042e7c4dfab8f85a169befca34 2023-04-22 14:56:11,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/e119dee04f114e9f9f8184387e1829e1 2023-04-22 14:56:11,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7121708895384e31b6f5feabe407ef80 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80 2023-04-22 14:56:11,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80, entries=350, sequenceid=60, filesize=52.3 K 2023-04-22 14:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/be8522042e7c4dfab8f85a169befca34 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34 2023-04-22 14:56:11,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34, entries=150, sequenceid=60, filesize=11.6 K 2023-04-22 14:56:11,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/e119dee04f114e9f9f8184387e1829e1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1 2023-04-22 14:56:11,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1, entries=150, sequenceid=60, filesize=11.6 K 2023-04-22 14:56:11,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 89f2897412e14388617caefccca798eb in 490ms, sequenceid=60, compaction requested=true 2023-04-22 14:56:11,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:11,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:11,712 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:11,712 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:11,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:11,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:11,713 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:11,713 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:11,713 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,713 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=102.5 K 2023-04-22 14:56:11,713 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:11,714 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:11,714 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 547361f3a5674052829477f012323880, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682175369044 2023-04-22 14:56:11,714 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,714 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=34.8 K 2023-04-22 14:56:11,714 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7623c3b769d94a0d8613ab5f35543c35, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682175369076 2023-04-22 14:56:11,714 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0fd69fb02f8f41a2945f643cb6439cda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682175369044 2023-04-22 14:56:11,715 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7121708895384e31b6f5feabe407ef80, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371197 2023-04-22 14:56:11,715 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a4b49550a4fe46a2bba0d79058a8af10, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682175369076 2023-04-22 14:56:11,715 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting be8522042e7c4dfab8f85a169befca34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371215 2023-04-22 14:56:11,722 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:11,724 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#518 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:11,734 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/d5fd403c78804b4e9b7897b5adbbd35f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/d5fd403c78804b4e9b7897b5adbbd35f 2023-04-22 14:56:11,739 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into d5fd403c78804b4e9b7897b5adbbd35f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:11,739 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:11,739 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=13, startTime=1682175371712; duration=0sec 2023-04-22 14:56:11,739 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:11,739 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:11,741 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:11,741 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:11,741 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,741 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=34.8 K 2023-04-22 14:56:11,741 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2c84140fd88c40199fd7808b4a239b48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682175369044 2023-04-22 14:56:11,742 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3a17af15104441c09cc16ccaa7c4135d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682175369076 2023-04-22 14:56:11,742 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e119dee04f114e9f9f8184387e1829e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371215 2023-04-22 14:56:11,748 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:11,749 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:11,750 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/e97129b9115f4e638df7422a09322ec1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e97129b9115f4e638df7422a09322ec1 2023-04-22 14:56:11,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:11,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:11,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:11,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:11,754 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:11,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:11,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:11,754 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into e97129b9115f4e638df7422a09322ec1(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:11,754 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:11,754 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=13, startTime=1682175371712; duration=0sec 2023-04-22 14:56:11,754 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,755 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:11,755 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,756 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:11,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:11,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:11,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:11,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:11,758 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:11,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:11,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:11,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:11,759 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:11,759 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:11,759 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:11,760 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:11,760 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/4179322398a94c8ba3438ca4425c227d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/4179322398a94c8ba3438ca4425c227d 2023-04-22 14:56:11,764 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into 4179322398a94c8ba3438ca4425c227d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:11,765 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:11,765 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=13, startTime=1682175371712; duration=0sec 2023-04-22 14:56:11,765 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:11,781 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422ef6e997927d64982a0d2c1b03b6cc100 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ef6e997927d64982a0d2c1b03b6cc100 2023-04-22 14:56:11,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:11,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:11,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175431877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175431880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175431880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175431880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175431881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175431985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175431985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:11,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175431985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:11,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175431985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175432188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175432188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175432188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175432189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,192 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/48461b99eeb54211a3187c12525618ee 2023-04-22 14:56:12,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175432383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175432491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175432491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175432494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175432500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,602 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/9aacb413678d4106b432f407cca065d3 2023-04-22 14:56:12,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175432995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:12,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:12,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175432997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175433001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175433005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,014 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/9bc137656bb7465bae1e94ede6764f5f 2023-04-22 14:56:13,019 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/48461b99eeb54211a3187c12525618ee as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee 2023-04-22 14:56:13,023 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee, entries=150, sequenceid=97, filesize=25.1 K 2023-04-22 14:56:13,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/9aacb413678d4106b432f407cca065d3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3 2023-04-22 14:56:13,027 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3, entries=150, sequenceid=97, filesize=11.6 K 2023-04-22 14:56:13,027 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/9bc137656bb7465bae1e94ede6764f5f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f 2023-04-22 14:56:13,030 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f, entries=150, sequenceid=97, filesize=11.6 K 2023-04-22 14:56:13,031 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=46.96 KB/48090 for 89f2897412e14388617caefccca798eb in 1272ms, sequenceid=97, compaction requested=false 2023-04-22 14:56:13,031 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:13,031 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:13,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:13,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:13,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:13,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d827[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:13,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:13,035 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,038 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:13,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:13,038 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:13,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:13,040 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:13,042 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:13,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:13,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:13,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:13,042 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:13,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,048 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:13,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:13,049 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,049 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,049 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:13,049 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:13,149 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:13,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:13,153 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,155 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,155 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:13,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:13,157 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,157 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:13,157 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:13,158 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:13,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:13,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:13,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175373428, max timestamp=1682175373444Num uniques 859; 2023-04-22 14:56:13,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:13,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175373428, max timestamp=1682175373444Num uniques 859; 2023-04-22 14:56:13,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:13,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175373428, max timestamp=1682175373444Num uniques 859; 2023-04-22 14:56:13,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:13,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175433508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,573 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422616953fefb554c84b2cd93ee4b31258d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422616953fefb554c84b2cd93ee4b31258d 2023-04-22 14:56:13,577 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7a21f2b683354971bc8c1163dfadb36a 2023-04-22 14:56:13,586 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/2cd633a1d1fd42c1b3c025d80c977f6d 2023-04-22 14:56:13,595 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/bcf2be7b582b41aa964f800dcbfe33ed 2023-04-22 14:56:13,599 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/7a21f2b683354971bc8c1163dfadb36a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a 2023-04-22 14:56:13,603 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a, entries=150, sequenceid=108, filesize=25.1 K 2023-04-22 14:56:13,603 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/2cd633a1d1fd42c1b3c025d80c977f6d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d 2023-04-22 14:56:13,606 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d, entries=150, sequenceid=108, filesize=11.6 K 2023-04-22 14:56:13,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/bcf2be7b582b41aa964f800dcbfe33ed as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed 2023-04-22 14:56:13,610 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed, entries=150, sequenceid=108, filesize=11.6 K 2023-04-22 14:56:13,611 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 89f2897412e14388617caefccca798eb in 454ms, sequenceid=108, compaction requested=true 2023-04-22 14:56:13,611 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:13,611 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:13,613 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,613 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-22 14:56:13,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:13,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:13,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:13,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:13,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:13,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:13,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b1a35d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:13,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:13,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,621 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:13,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,623 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:13,624 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:13,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:13,627 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:13,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:13,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:13,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:13,627 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:13,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,632 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,634 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042286ab6e58b91948e9b5ea1a1caab3c19a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042286ab6e58b91948e9b5ea1a1caab3c19a 2023-04-22 14:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,636 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:13,636 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:13,648 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c2c2001c5bd1456cb9dab8868f9ce404 2023-04-22 14:56:13,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175433646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/a0b2e599e8f449d690cbd73314afc9d9 2023-04-22 14:56:13,737 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:13,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:13,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:13,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:13,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:13,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:13,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:13,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:13,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:13,743 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,743 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:13,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:13,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:13,749 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,749 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,750 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e809cd1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:13,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:13,751 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175433749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,752 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:13,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:13,753 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:13,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,753 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:13,754 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:13,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:13,756 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:13,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:13,756 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:13,757 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:13,757 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:13,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,758 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,759 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,760 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,764 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:13,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,764 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,764 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:13,764 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:13,864 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:13,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:13,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:13,868 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:13,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:13,868 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:13,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,869 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,870 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,870 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,870 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,870 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:13,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:13,872 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,872 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:13,872 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:13,872 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:13,874 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,875 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3087b2f7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:13,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:13,876 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,877 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:13,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:13,877 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:13,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,878 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:13,880 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:13,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:13,882 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:13,882 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:13,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:13,883 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:13,883 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:13,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:13,884 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:13,896 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,898 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:13,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:13,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:13,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:13,900 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:13,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:13,901 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:13,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:13,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:13,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:13,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:13,901 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:13,901 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:13,901 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:13,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:13,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:13,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44956 deadline: 1682175433953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,001 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:14,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:14,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:14,005 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:14,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:14,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:14,005 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:14,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,006 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175434004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,007 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:14,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:14,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,007 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:14,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:14,009 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,009 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:14,009 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:14,009 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:14,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:14,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175434007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,012 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,012 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:14,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,013 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec99420[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:14,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:14,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,014 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175434011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,015 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,015 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:14,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,016 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:14,017 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175434016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:14,019 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:14,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:14,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:14,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:14,020 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:14,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:14,021 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:14,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,023 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:14,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:14,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:14,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,027 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,027 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:14,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:14,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:14,026 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:14,027 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,027 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:14,027 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,027 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:14,027 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:14,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:14,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3f0bdae1f5f749c0a09fe33ddae184e0 2023-04-22 14:56:14,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c2c2001c5bd1456cb9dab8868f9ce404 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404 2023-04-22 14:56:14,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404, entries=200, sequenceid=146, filesize=32.1 K 2023-04-22 14:56:14,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/a0b2e599e8f449d690cbd73314afc9d9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9 2023-04-22 14:56:14,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9, entries=150, sequenceid=146, filesize=11.8 K 2023-04-22 14:56:14,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3f0bdae1f5f749c0a09fe33ddae184e0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0 2023-04-22 14:56:14,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0, entries=150, sequenceid=146, filesize=11.8 K 2023-04-22 14:56:14,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 89f2897412e14388617caefccca798eb in 499ms, sequenceid=146, compaction requested=true 2023-04-22 14:56:14,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:14,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:14,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:14,114 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:14,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:14,114 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:14,115 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110227 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:14,115 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:14,115 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,115 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e97129b9115f4e638df7422a09322ec1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=107.6 K 2023-04-22 14:56:14,116 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:14,116 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e97129b9115f4e638df7422a09322ec1, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371215 2023-04-22 14:56:14,116 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:14,116 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,116 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/d5fd403c78804b4e9b7897b5adbbd35f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=46.7 K 2023-04-22 14:56:14,116 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 48461b99eeb54211a3187c12525618ee, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682175371256 2023-04-22 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d5fd403c78804b4e9b7897b5adbbd35f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371215 2023-04-22 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 7a21f2b683354971bc8c1163dfadb36a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682175371875 2023-04-22 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9aacb413678d4106b432f407cca065d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682175371256 2023-04-22 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c2c2001c5bd1456cb9dab8868f9ce404, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2cd633a1d1fd42c1b3c025d80c977f6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682175371875 2023-04-22 14:56:14,118 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting a0b2e599e8f449d690cbd73314afc9d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:14,127 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:14,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:14,129 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:14,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:14,138 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#531 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:14,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:14,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:14,149 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:14,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:14,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:14,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:14,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:14,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:14,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:14,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:14,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:14,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:14,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:14,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:14,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:14,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:14,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:14,162 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,162 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:14,165 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:56:14,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:14,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:14,165 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:14,166 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:14,178 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422e43afd12c27945fdbdceb685ac13f998 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e43afd12c27945fdbdceb685ac13f998 2023-04-22 14:56:14,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:14,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175374299, max timestamp=1682175374322Num uniques 859; 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175374299, max timestamp=1682175374322Num uniques 859; 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175374299, max timestamp=1682175374322Num uniques 859; 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:14,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175374363, max timestamp=1682175374380Num uniques 1018; 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175374363, max timestamp=1682175374380Num uniques 1018; 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175374363, max timestamp=1682175374380Num uniques 1018; 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175374429, max timestamp=1682175374447Num uniques 1027; 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175374429, max timestamp=1682175374447Num uniques 1027; 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682175374429, max timestamp=1682175374447Num uniques 1027; 2023-04-22 14:56:14,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175374493, max timestamp=1682175374533Num uniques 1036; 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175374493, max timestamp=1682175374533Num uniques 1036; 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682175374493, max timestamp=1682175374533Num uniques 1036; 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:56:14,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-22 14:56:14,578 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/29d6fa2e01584cdab177d5eb59dffa05 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/29d6fa2e01584cdab177d5eb59dffa05 2023-04-22 14:56:14,583 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3ee83f39e78e444f9e4418b49113ad96 2023-04-22 14:56:14,585 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into 29d6fa2e01584cdab177d5eb59dffa05(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:14,585 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:14,585 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=12, startTime=1682175374114; duration=0sec 2023-04-22 14:56:14,586 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:14,586 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:14,588 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:14,588 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:14,589 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:14,589 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/4179322398a94c8ba3438ca4425c227d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=46.7 K 2023-04-22 14:56:14,589 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4179322398a94c8ba3438ca4425c227d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682175371215 2023-04-22 14:56:14,590 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9bc137656bb7465bae1e94ede6764f5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682175371256 2023-04-22 14:56:14,590 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/55c875301ec24e4a925ce578f8d0d460 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/55c875301ec24e4a925ce578f8d0d460 2023-04-22 14:56:14,590 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting bcf2be7b582b41aa964f800dcbfe33ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682175371875 2023-04-22 14:56:14,590 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3f0bdae1f5f749c0a09fe33ddae184e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:14,598 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into 55c875301ec24e4a925ce578f8d0d460(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:14,598 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:14,598 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=12, startTime=1682175374114; duration=0sec 2023-04-22 14:56:14,598 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:14,603 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:14,610 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/2c6f93bf095e43fea35bdfa7c6888504 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c6f93bf095e43fea35bdfa7c6888504 2023-04-22 14:56:14,614 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into 2c6f93bf095e43fea35bdfa7c6888504(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:14,614 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:14,614 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=12, startTime=1682175374114; duration=0sec 2023-04-22 14:56:14,614 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:14,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175374604, max timestamp=1682175374619Num uniques 1045; 2023-04-22 14:56:14,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:56:14,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175374604, max timestamp=1682175374619Num uniques 1045; 2023-04-22 14:56:14,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682175374604, max timestamp=1682175374619Num uniques 1045; 2023-04-22 14:56:14,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:56:14,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-22 14:56:14,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175434666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175434768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:14,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175434971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:14,997 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/63cf96732b874847b6f9ec500b74547e 2023-04-22 14:56:15,007 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/497329bd39ce4957a99720fbb637cd9a 2023-04-22 14:56:15,010 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3ee83f39e78e444f9e4418b49113ad96 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96 2023-04-22 14:56:15,013 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96, entries=100, sequenceid=155, filesize=18.5 K 2023-04-22 14:56:15,014 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/63cf96732b874847b6f9ec500b74547e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e 2023-04-22 14:56:15,024 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e, entries=100, sequenceid=155, filesize=9.4 K 2023-04-22 14:56:15,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/497329bd39ce4957a99720fbb637cd9a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a 2023-04-22 14:56:15,028 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a, entries=100, sequenceid=155, filesize=9.4 K 2023-04-22 14:56:15,029 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 89f2897412e14388617caefccca798eb in 864ms, sequenceid=155, compaction requested=false 2023-04-22 14:56:15,029 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:15,029 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:15,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:15,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419d63ef[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:15,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:15,033 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,034 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:15,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:15,039 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,039 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,039 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,039 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:15,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:15,042 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:15,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:15,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:15,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:15,042 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:15,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,044 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,045 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,051 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:15,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:15,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:15,051 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:15,052 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:15,152 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:15,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:15,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:15,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:15,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:15,156 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:15,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:15,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:15,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:15,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:15,159 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:15,159 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,159 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:15,159 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,160 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:15,160 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=140.89 KB heapSize=401.53 KB 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:15,160 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:15,176 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042274f3ba8228d845c997b218e2de461bda to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042274f3ba8228d845c997b218e2de461bda 2023-04-22 14:56:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:15,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:15,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:15,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175435290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:15,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175435392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,582 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3f37691557c245aea3d2d324f22302e3 2023-04-22 14:56:15,595 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/4532cf61a85b4b1fa6766baf97007c18 2023-04-22 14:56:15,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:15,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175435595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,603 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/1f830b28e35245f7830a721668fe7423 2023-04-22 14:56:15,607 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3f37691557c245aea3d2d324f22302e3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3 2023-04-22 14:56:15,611 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3, entries=150, sequenceid=264, filesize=25.4 K 2023-04-22 14:56:15,611 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/4532cf61a85b4b1fa6766baf97007c18 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18 2023-04-22 14:56:15,615 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18, entries=150, sequenceid=264, filesize=11.9 K 2023-04-22 14:56:15,615 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/1f830b28e35245f7830a721668fe7423 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423 2023-04-22 14:56:15,619 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423, entries=150, sequenceid=264, filesize=11.9 K 2023-04-22 14:56:15,619 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~401.48 KB/411120, currentSize=40.25 KB/41220 for 89f2897412e14388617caefccca798eb in 459ms, sequenceid=264, compaction requested=true 2023-04-22 14:56:15,619 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:15,620 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:15,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b111919[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:15,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:15,624 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,625 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:15,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:15,625 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:15,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:15,627 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,627 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:15,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:15,629 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:15,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:15,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:15,629 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:15,629 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:15,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,631 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,632 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,633 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:15,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:15,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,636 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,636 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,637 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:15,637 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:15,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,637 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,737 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:15,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:15,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:15,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:15,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:15,740 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:15,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:15,740 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:15,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,741 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,742 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,742 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:15,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:15,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:15,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:15,743 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:15,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:15,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:15,744 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:15,744 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:56:15,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:15,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,744 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:15,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:15,745 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,769 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d016633edc264187a5649238533afd87 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d016633edc264187a5649238533afd87 2023-04-22 14:56:15,776 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=273, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/76209e07868c4218a04317bb1f7d5d76 2023-04-22 14:56:15,794 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c3cb80a4827147ee9736b38139b67c10 2023-04-22 14:56:15,806 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/71d758fe948c4a87a2909498b94108ff 2023-04-22 14:56:15,810 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/76209e07868c4218a04317bb1f7d5d76 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76 2023-04-22 14:56:15,813 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76, entries=150, sequenceid=273, filesize=25.4 K 2023-04-22 14:56:15,814 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c3cb80a4827147ee9736b38139b67c10 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10 2023-04-22 14:56:15,818 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10, entries=150, sequenceid=273, filesize=11.9 K 2023-04-22 14:56:15,818 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/71d758fe948c4a87a2909498b94108ff as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff 2023-04-22 14:56:15,822 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff, entries=150, sequenceid=273, filesize=11.9 K 2023-04-22 14:56:15,822 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 89f2897412e14388617caefccca798eb in 78ms, sequenceid=273, compaction requested=true 2023-04-22 14:56:15,822 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:15,823 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:15,823 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,824 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,825 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260c1686[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:15,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:15,827 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,828 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:15,828 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,828 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:15,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:15,828 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:15,830 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:15,833 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:15,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:15,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:15,833 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:15,833 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:15,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,835 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:15,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,836 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,837 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:15,840 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:15,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:15,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:15,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,840 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:15,840 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:15,839 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:15,840 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,840 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,840 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:15,840 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:15,940 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:15,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:15,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:15,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:15,944 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:15,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:15,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:15,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,944 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,951 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:15,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:15,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:15,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:15,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:15,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:15,954 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:15,954 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:56:15,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:15,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:15,955 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:15,965 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422c9c640a5c48c408faca0591ae9985d86 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9c640a5c48c408faca0591ae9985d86 2023-04-22 14:56:15,974 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/ab17a8db22184982b7ded27b797d0760 2023-04-22 14:56:15,991 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/86743c5ead3a43aeb836581096474f75 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175376005, max timestamp=1682175376016Num uniques 859; 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175376005, max timestamp=1682175376016Num uniques 859; 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175376005, max timestamp=1682175376016Num uniques 859; 2023-04-22 14:56:16,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:16,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,402 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/38327b981af84c4f991708525181553e 2023-04-22 14:56:16,405 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/ab17a8db22184982b7ded27b797d0760 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760 2023-04-22 14:56:16,408 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760, entries=350, sequenceid=283, filesize=52.9 K 2023-04-22 14:56:16,409 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/86743c5ead3a43aeb836581096474f75 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75 2023-04-22 14:56:16,412 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75, entries=100, sequenceid=283, filesize=9.5 K 2023-04-22 14:56:16,413 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/38327b981af84c4f991708525181553e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e 2023-04-22 14:56:16,416 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e, entries=100, sequenceid=283, filesize=9.5 K 2023-04-22 14:56:16,417 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 89f2897412e14388617caefccca798eb in 463ms, sequenceid=283, compaction requested=true 2023-04-22 14:56:16,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:16,417 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,418 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5496fef3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:16,420 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:16,420 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,422 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:16,422 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,422 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:16,423 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,423 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:16,426 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:16,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:16,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:16,426 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:16,426 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:16,431 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,431 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:16,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,431 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:16,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:16,431 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:16,433 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:16,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:16,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:16,435 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:16,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:16,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:16,435 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:16,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:16,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:16,436 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:16,436 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:16,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:16,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:16,436 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,536 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:16,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:16,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:16,539 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:16,539 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:16,539 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:16,539 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:16,540 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,540 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:16,541 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:16,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:16,541 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:16,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,542 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:16,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:16,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:16,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:16,543 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:16,543 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:16,544 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:16,544 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:16,557 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042279ee8238a0634cdf96deb3144089a93a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042279ee8238a0634cdf96deb3144089a93a 2023-04-22 14:56:16,565 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/e2caf953ac224ceba487a425ed83e8f5 2023-04-22 14:56:16,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:16,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:16,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175436960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175436960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175436960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175436960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175436960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:16,975 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/e67af1b17dcb4f84a7e70c6d6898db74 2023-04-22 14:56:17,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175437265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175437265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175437265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175437265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175437265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,390 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/f40a64e7acaf4ccd8ebd9821e58127ef 2023-04-22 14:56:17,394 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/e2caf953ac224ceba487a425ed83e8f5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5 2023-04-22 14:56:17,398 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5, entries=150, sequenceid=319, filesize=25.4 K 2023-04-22 14:56:17,399 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/e67af1b17dcb4f84a7e70c6d6898db74 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74 2023-04-22 14:56:17,402 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74, entries=150, sequenceid=319, filesize=11.9 K 2023-04-22 14:56:17,403 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/f40a64e7acaf4ccd8ebd9821e58127ef as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef 2023-04-22 14:56:17,406 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef, entries=150, sequenceid=319, filesize=11.9 K 2023-04-22 14:56:17,407 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 89f2897412e14388617caefccca798eb in 864ms, sequenceid=319, compaction requested=true 2023-04-22 14:56:17,407 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:17,407 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:17,409 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,409 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:17,409 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,409 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:17,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:17,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebe6d95[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:17,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:17,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,413 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,413 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:17,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:17,413 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:17,414 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:17,415 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:17,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:17,415 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:17,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:17,418 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:17,418 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:17,418 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:17,418 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:17,418 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:17,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:17,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,419 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:17,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:17,420 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,421 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,423 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:17,423 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:17,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:17,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:17,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:17,423 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:17,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:17,424 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:17,424 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:17,423 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:17,424 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,524 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:17,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:17,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:17,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:17,528 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:17,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:17,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:17,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:17,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,528 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:17,530 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:17,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:17,530 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:17,531 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,531 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:17,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:17,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:17,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:17,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:17,532 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:17,532 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:17,532 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:56:17,532 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:17,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:17,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:17,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:17,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:17,533 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:17,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:17,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175437786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175437787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175377785, max timestamp=1682175377789Num uniques 859; 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175377785, max timestamp=1682175377789Num uniques 859; 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682175377785, max timestamp=1682175377789Num uniques 859; 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:17,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-22 14:56:17,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175437816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175437823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175437824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175437891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175437892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175437925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175437930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175437931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:17,950 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422be1e64a37a374a638a06253438ad7f59 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422be1e64a37a374a638a06253438ad7f59 2023-04-22 14:56:18,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175438095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175438097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175438130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175438138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175438138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,361 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/bfbed64c287f455f8fca1e7e68f66cf3 2023-04-22 14:56:18,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175438399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175438401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175438436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175438441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175438441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,771 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0 2023-04-22 14:56:18,793 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/73f1f69ec7e7422793020f126631ac99 2023-04-22 14:56:18,798 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/bfbed64c287f455f8fca1e7e68f66cf3 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3 2023-04-22 14:56:18,802 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3, entries=150, sequenceid=329, filesize=25.4 K 2023-04-22 14:56:18,803 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0 2023-04-22 14:56:18,806 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0, entries=150, sequenceid=329, filesize=11.9 K 2023-04-22 14:56:18,807 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/73f1f69ec7e7422793020f126631ac99 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99 2023-04-22 14:56:18,810 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99, entries=150, sequenceid=329, filesize=11.9 K 2023-04-22 14:56:18,811 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 89f2897412e14388617caefccca798eb in 1279ms, sequenceid=329, compaction requested=true 2023-04-22 14:56:18,811 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:18,811 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:18,813 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:18,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d447a8a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:18,814 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:18,814 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,815 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:18,816 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:18,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,816 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:18,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:18,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:18,817 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,817 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:18,819 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:18,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:18,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:18,819 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:18,819 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:18,822 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,822 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:18,822 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,823 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,824 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:18,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:18,826 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,826 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,826 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:18,826 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:18,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:18,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-22 14:56:18,921 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304225fdd43a9d2d44eeeba32e5c969a29e30 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225fdd43a9d2d44eeeba32e5c969a29e30 2023-04-22 14:56:18,926 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:18,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:18,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175438923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175438924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:18,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:18,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:18,931 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:18,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:18,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:18,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:18,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,931 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,933 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,933 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:18,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,934 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:18,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:18,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:18,934 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:18,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:18,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:18,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:18,935 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:18,937 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,937 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,937 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,937 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3aff82[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:18,939 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:18,939 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,942 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:18,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,942 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:18,943 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:18,944 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175438945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:18,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:18,947 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:18,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:18,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:18,947 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:18,947 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:18,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,949 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:18,950 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,951 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175438949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175438951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,957 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:18,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:18,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:18,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:18,958 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:18,958 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:18,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:18,958 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:18,958 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175439029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175439029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,058 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:19,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:19,062 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:19,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:19,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:19,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,062 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,064 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,064 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:19,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:19,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,065 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:19,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:19,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:19,066 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,067 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:19,067 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:19,068 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,068 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b8fb70[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:19,069 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:19,070 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,070 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:19,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:19,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,071 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:19,071 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,073 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:19,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:19,076 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:19,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:19,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:19,076 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:19,076 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:19,077 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,077 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,077 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,078 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,080 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,082 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,082 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,082 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,082 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,083 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:19,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:19,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,083 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,083 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,083 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:19,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,183 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:19,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:19,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:19,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:19,187 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:19,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,187 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:19,188 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:19,188 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,188 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,189 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,189 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,190 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:19,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:19,191 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,191 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:19,191 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:19,192 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,194 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7567f337[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:19,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:19,196 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,197 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:19,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,198 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:19,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:19,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,199 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:19,199 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:19,202 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:19,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:19,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:19,202 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:19,202 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:19,204 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,204 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,204 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,205 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,206 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,209 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,209 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:19,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:19,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,210 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:19,210 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:19,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175439235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175439236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,310 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:19,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:19,314 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:19,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:19,314 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:19,315 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,315 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,316 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,316 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:19,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:19,319 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b09573b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:19,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:19,321 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,322 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:19,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,323 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,325 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,327 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3a25649eeeab4df5a7722cdeeaf4050d 2023-04-22 14:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:19,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:19,328 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:19,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:19,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:19,328 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:19,328 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:19,330 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,331 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,331 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,331 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,334 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:19,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:19,339 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,340 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:19,340 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:19,339 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,340 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,340 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/0be616ab79e849f8abe08195fbaef3f2 2023-04-22 14:56:19,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:19,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/09cd9c80c1d24744b348af99680817aa 2023-04-22 14:56:19,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3a25649eeeab4df5a7722cdeeaf4050d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d 2023-04-22 14:56:19,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d, entries=200, sequenceid=367, filesize=32.3 K 2023-04-22 14:56:19,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/0be616ab79e849f8abe08195fbaef3f2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2 2023-04-22 14:56:19,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2, entries=150, sequenceid=367, filesize=11.9 K 2023-04-22 14:56:19,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/09cd9c80c1d24744b348af99680817aa as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa 2023-04-22 14:56:19,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa, entries=150, sequenceid=367, filesize=11.9 K 2023-04-22 14:56:19,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 89f2897412e14388617caefccca798eb in 462ms, sequenceid=367, compaction requested=true 2023-04-22 14:56:19,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:19,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:19,370 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 14:56:19,370 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 14:56:19,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:19,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:19,372 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 236514 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 14:56:19,373 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 92590 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 14:56:19,373 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:19,373 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:19,373 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,373 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,373 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/55c875301ec24e4a925ce578f8d0d460, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=231.0 K 2023-04-22 14:56:19,373 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/29d6fa2e01584cdab177d5eb59dffa05, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=90.4 K 2023-04-22 14:56:19,373 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 55c875301ec24e4a925ce578f8d0d460, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:19,373 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 29d6fa2e01584cdab177d5eb59dffa05, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3ee83f39e78e444f9e4418b49113ad96, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682175373630 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 63cf96732b874847b6f9ec500b74547e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682175373630 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3f37691557c245aea3d2d324f22302e3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682175374656 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 4532cf61a85b4b1fa6766baf97007c18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682175374656 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 76209e07868c4218a04317bb1f7d5d76, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682175375276 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c3cb80a4827147ee9736b38139b67c10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682175375276 2023-04-22 14:56:19,374 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting ab17a8db22184982b7ded27b797d0760, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682175375915 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 86743c5ead3a43aeb836581096474f75, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682175375939 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e2caf953ac224ceba487a425ed83e8f5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682175376032 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e67af1b17dcb4f84a7e70c6d6898db74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682175376032 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting bfbed64c287f455f8fca1e7e68f66cf3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682175376647 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 81a1ce8e0e8d49c3ab9ffbb5d3ba92e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682175376647 2023-04-22 14:56:19,375 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3a25649eeeab4df5a7722cdeeaf4050d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:19,376 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0be616ab79e849f8abe08195fbaef3f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:19,391 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#555 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:19,392 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#554 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:19,403 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/3b6d93ad5ee64b168d1ea85c87815134 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3b6d93ad5ee64b168d1ea85c87815134 2023-04-22 14:56:19,407 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into 3b6d93ad5ee64b168d1ea85c87815134(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:19,407 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:19,407 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=8, startTime=1682175379370; duration=0sec 2023-04-22 14:56:19,407 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:19,407 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-22 14:56:19,409 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 92590 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-22 14:56:19,409 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:19,409 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,409 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c6f93bf095e43fea35bdfa7c6888504, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=90.4 K 2023-04-22 14:56:19,410 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2c6f93bf095e43fea35bdfa7c6888504, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682175373460 2023-04-22 14:56:19,410 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 497329bd39ce4957a99720fbb637cd9a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682175373630 2023-04-22 14:56:19,410 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 1f830b28e35245f7830a721668fe7423, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682175374656 2023-04-22 14:56:19,411 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 71d758fe948c4a87a2909498b94108ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682175375276 2023-04-22 14:56:19,411 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 38327b981af84c4f991708525181553e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682175375939 2023-04-22 14:56:19,411 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting f40a64e7acaf4ccd8ebd9821e58127ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682175376032 2023-04-22 14:56:19,411 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 73f1f69ec7e7422793020f126631ac99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682175376647 2023-04-22 14:56:19,412 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 09cd9c80c1d24744b348af99680817aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:19,422 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#556 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:19,440 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:19,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:19,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:19,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:19,443 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:19,443 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,443 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:19,444 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:19,444 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,444 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,447 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,447 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,447 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,447 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:19,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:19,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:19,449 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:19,450 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:19,450 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:19,465 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422b949184a509d4321b5665de476a5019c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b949184a509d4321b5665de476a5019c 2023-04-22 14:56:19,470 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/d5686fc30b9d491db602a825528c4af1 2023-04-22 14:56:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:19,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175379569, max timestamp=1682175379581Num uniques 859; 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175379569, max timestamp=1682175379581Num uniques 859; 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175379569, max timestamp=1682175379581Num uniques 859; 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:19,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:19,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175439617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175439618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175439722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175439723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,835 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/790ba520ce0d4a35b68600fc05d75708 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/790ba520ce0d4a35b68600fc05d75708 2023-04-22 14:56:19,839 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into 790ba520ce0d4a35b68600fc05d75708(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:19,839 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:19,839 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=8, startTime=1682175379371; duration=0sec 2023-04-22 14:56:19,839 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:19,878 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/748640e8ab484947b3e7cfe0e77230f7 2023-04-22 14:56:19,888 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/e38b222bc5d2416ea11f24f46bca14dd 2023-04-22 14:56:19,892 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/d5686fc30b9d491db602a825528c4af1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1 2023-04-22 14:56:19,896 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1, entries=100, sequenceid=377, filesize=18.6 K 2023-04-22 14:56:19,897 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/748640e8ab484947b3e7cfe0e77230f7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7 2023-04-22 14:56:19,900 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7, entries=100, sequenceid=377, filesize=9.5 K 2023-04-22 14:56:19,901 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/e38b222bc5d2416ea11f24f46bca14dd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd 2023-04-22 14:56:19,904 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd, entries=100, sequenceid=377, filesize=9.5 K 2023-04-22 14:56:19,905 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 89f2897412e14388617caefccca798eb in 455ms, sequenceid=377, compaction requested=false 2023-04-22 14:56:19,905 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:19,905 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:19,907 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:19,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47037f37[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:19,909 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:19,909 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,910 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:19,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:19,910 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:19,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:19,911 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:19,914 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:19,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:19,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:19,914 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:19,914 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:19,921 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,921 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,921 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:19,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,922 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,923 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:19,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:19,925 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:19,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:19,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:19,925 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,926 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:19,926 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:19,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:19,926 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:19,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175379600, max timestamp=1682175379928Num uniques 1068; 2023-04-22 14:56:19,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:56:19,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175379600, max timestamp=1682175379928Num uniques 1068; 2023-04-22 14:56:19,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:56:19,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682175379600, max timestamp=1682175379928Num uniques 1068; 2023-04-22 14:56:19,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-22 14:56:19,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:19,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:19,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:19,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175439962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175439963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175379961, max timestamp=1682175379966Num uniques 859; 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175379961, max timestamp=1682175379966Num uniques 859; 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682175379961, max timestamp=1682175379966Num uniques 859; 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-22 14:56:19,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175439988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175439991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:19,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:19,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175439994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,026 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:20,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:20,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:20,030 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:20,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:20,030 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:20,031 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,031 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,032 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,032 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:20,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:20,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,034 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:20,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:20,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:20,035 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:20,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,037 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,037 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,037 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:20,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785f7fdc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:20,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:20,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:20,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:20,043 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:20,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:20,046 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:20,046 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:20,046 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:20,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,048 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,049 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,050 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,053 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,053 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:20,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:20,054 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:20,054 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175440069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175440070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175440095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175440095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175440099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,154 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:20,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:20,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:20,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:20,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:20,158 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:20,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:20,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:20,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,158 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:20,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:20,163 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:20,164 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326bd5ab[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:20,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:20,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,167 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:20,167 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,168 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:20,167 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:20,169 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:20,171 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:20,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:20,171 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:20,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:20,172 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:20,174 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,174 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,174 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,175 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,176 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,178 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:20,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,178 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,179 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:20,179 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,179 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:20,179 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:20,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44908 deadline: 1682175440273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,279 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:20,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175440275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:20,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:20,282 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:20,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:20,282 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:20,283 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,283 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,284 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,284 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,284 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,284 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:20,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:20,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,285 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:20,286 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,286 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:20,286 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:20,286 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,291 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,291 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739aec52[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:20,293 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:20,293 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,294 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:20,294 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:20,294 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,296 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:20,299 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:20,299 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:20,299 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:20,299 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:20,299 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:20,300 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,300 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:20,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,301 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:20,301 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:20,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175440298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,303 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:20,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175440299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175440303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,307 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:20,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:20,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:20,307 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:20,307 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:20,308 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:20,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:20,358 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f81f25567175457fa3f77d0df955fce6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f81f25567175457fa3f77d0df955fce6 2023-04-22 14:56:20,364 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=421, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/9a9510dec03542dd95c8c09ee5d67d52 2023-04-22 14:56:20,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/56298b24f7a94ddc9c83db50d9c3c5a6 2023-04-22 14:56:20,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/772d940a85ba483ab9bb103009330091 2023-04-22 14:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/9a9510dec03542dd95c8c09ee5d67d52 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52 2023-04-22 14:56:20,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52, entries=200, sequenceid=421, filesize=32.3 K 2023-04-22 14:56:20,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/56298b24f7a94ddc9c83db50d9c3c5a6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6 2023-04-22 14:56:20,408 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:20,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:20,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:20,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:20,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:20,411 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:20,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:20,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:20,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,411 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,413 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:20,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:20,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,413 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:20,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:20,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6, entries=150, sequenceid=421, filesize=11.9 K 2023-04-22 14:56:20,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/772d940a85ba483ab9bb103009330091 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091 2023-04-22 14:56:20,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:20,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091, entries=150, sequenceid=421, filesize=11.9 K 2023-04-22 14:56:20,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.95 KB/168912, currentSize=127.47 KB/130530 for 89f2897412e14388617caefccca798eb in 491ms, sequenceid=421, compaction requested=true 2023-04-22 14:56:20,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:20,430 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 8 compacting, 2 eligible, 16 blocking 2023-04-22 14:56:20,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:20,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:20,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 14:56:20,431 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-22 14:56:20,431 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-22 14:56:20,431 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. because compaction request was cancelled 2023-04-22 14:56:20,431 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:20,432 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:20,432 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:20,432 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,432 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3b6d93ad5ee64b168d1ea85c87815134, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=33.8 K 2023-04-22 14:56:20,432 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3b6d93ad5ee64b168d1ea85c87815134, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:20,433 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 748640e8ab484947b3e7cfe0e77230f7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682175378922 2023-04-22 14:56:20,433 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 56298b24f7a94ddc9c83db50d9c3c5a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1682175379934 2023-04-22 14:56:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:20,444 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:20,444 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:20,444 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:20,445 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#563 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:20,445 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:20,465 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/e97946074467444389042ab53c680c6a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e97946074467444389042ab53c680c6a 2023-04-22 14:56:20,469 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into e97946074467444389042ab53c680c6a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:20,469 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:20,469 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=13, startTime=1682175380431; duration=0sec 2023-04-22 14:56:20,470 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:20,470 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-22 14:56:20,470 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-22 14:56:20,471 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:20,471 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:20,471 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/790ba520ce0d4a35b68600fc05d75708, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=33.8 K 2023-04-22 14:56:20,471 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 790ba520ce0d4a35b68600fc05d75708, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:20,472 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e38b222bc5d2416ea11f24f46bca14dd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682175378922 2023-04-22 14:56:20,472 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 772d940a85ba483ab9bb103009330091, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1682175379934 2023-04-22 14:56:20,479 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#565 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:20,491 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/0abd5e8a88114e03806ad6a4914444eb as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0abd5e8a88114e03806ad6a4914444eb 2023-04-22 14:56:20,497 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into 0abd5e8a88114e03806ad6a4914444eb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:20,497 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:20,497 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=13, startTime=1682175380431; duration=0sec 2023-04-22 14:56:20,497 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:20,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:20,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44928 deadline: 1682175440595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175440596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175440605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175440605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175440609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,614 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/f6afd0f2337b45db9ba2b5198a6ebc89 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f6afd0f2337b45db9ba2b5198a6ebc89 2023-04-22 14:56:20,618 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into f6afd0f2337b45db9ba2b5198a6ebc89(size=25.9 K), total size for store is 76.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-22 14:56:20,618 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:20,618 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=8, startTime=1682175379370; duration=1sec 2023-04-22 14:56:20,618 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175440699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175440699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,877 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422f82f1dc8eaaa418399381f6f65c6e5d2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f82f1dc8eaaa418399381f6f65c6e5d2 2023-04-22 14:56:20,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175440903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:20,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175440903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175441109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175441110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175441116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175441205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175441206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,283 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=455, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3362d16bbe554efbb50eeab6891f36ca 2023-04-22 14:56:21,694 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c46a524bc9904a8c859050793b2c3c5c 2023-04-22 14:56:21,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175441711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:21,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:21,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175441713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175442116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,122 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/d2b97b61b3f4486180b4cd7569e220e0 2023-04-22 14:56:22,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175442120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175442120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,126 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3362d16bbe554efbb50eeab6891f36ca as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca 2023-04-22 14:56:22,129 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca, entries=150, sequenceid=455, filesize=25.4 K 2023-04-22 14:56:22,130 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c46a524bc9904a8c859050793b2c3c5c as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c 2023-04-22 14:56:22,138 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c, entries=150, sequenceid=455, filesize=11.9 K 2023-04-22 14:56:22,138 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/d2b97b61b3f4486180b4cd7569e220e0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0 2023-04-22 14:56:22,141 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0, entries=150, sequenceid=455, filesize=11.9 K 2023-04-22 14:56:22,142 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 89f2897412e14388617caefccca798eb in 1698ms, sequenceid=455, compaction requested=true 2023-04-22 14:56:22,142 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:22,142 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:22,142 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbcdfd1[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:22,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:22,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:22,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:22,149 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:22,152 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:22,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:22,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:22,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:22,152 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:22,159 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,159 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:22,159 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:22,161 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:22,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:22,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:22,164 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:22,164 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,164 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:22,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:22,164 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:22,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:22,264 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:22,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:22,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:22,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:22,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:22,268 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:22,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:22,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:22,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:22,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,268 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:22,269 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:22,269 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:22,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:22,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:22,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:22,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:22,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:22,271 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:22,272 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:22,272 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:22,292 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224d52a5938d964145a9243bbf65a06dd4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224d52a5938d964145a9243bbf65a06dd4 2023-04-22 14:56:22,297 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=469, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/aef5f1757ff3428393fe1ee3842d22e6 2023-04-22 14:56:22,705 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/3f7f40ebf1aa47fead6bc199dd946d73 2023-04-22 14:56:22,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:22,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:22,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175382738, max timestamp=1682175382746Num uniques 859; 2023-04-22 14:56:22,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:22,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175382738, max timestamp=1682175382746Num uniques 859; 2023-04-22 14:56:22,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:22,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175382738, max timestamp=1682175382746Num uniques 859; 2023-04-22 14:56:22,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:22,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175442806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175442807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175442913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:22,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:22,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175442913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,115 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/b7699effca544ea2bc9742d6293aec08 2023-04-22 14:56:23,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175443116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175443117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,120 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/aef5f1757ff3428393fe1ee3842d22e6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6 2023-04-22 14:56:23,124 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6, entries=150, sequenceid=469, filesize=25.4 K 2023-04-22 14:56:23,125 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/3f7f40ebf1aa47fead6bc199dd946d73 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73 2023-04-22 14:56:23,131 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73, entries=150, sequenceid=469, filesize=11.9 K 2023-04-22 14:56:23,131 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/b7699effca544ea2bc9742d6293aec08 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08 2023-04-22 14:56:23,134 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08, entries=150, sequenceid=469, filesize=11.9 K 2023-04-22 14:56:23,135 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 89f2897412e14388617caefccca798eb in 863ms, sequenceid=469, compaction requested=true 2023-04-22 14:56:23,135 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:23,135 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,137 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:23,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1ef465[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:23,139 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:23,139 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,141 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:23,141 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,142 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:23,143 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:23,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:23,146 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:23,146 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:23,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,147 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,149 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,152 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:23,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,152 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:23,152 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:23,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,153 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,153 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,153 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,253 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:23,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:23,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:23,257 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,259 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,260 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:23,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:23,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:23,262 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:23,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:23,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175443433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175443437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175443538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175443540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,677 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304224eea908598154d07af0b39535f6c2ad8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224eea908598154d07af0b39535f6c2ad8 2023-04-22 14:56:23,683 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=504, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/4100718db34d481e94f4ac0bb7c7b760 2023-04-22 14:56:23,692 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/8a6e5293c2d745edbadbf852ab9d5fd4 2023-04-22 14:56:23,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3042a20a2ab24d13820e776861ca0435 2023-04-22 14:56:23,703 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/4100718db34d481e94f4ac0bb7c7b760 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760 2023-04-22 14:56:23,706 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760, entries=150, sequenceid=504, filesize=25.4 K 2023-04-22 14:56:23,707 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/8a6e5293c2d745edbadbf852ab9d5fd4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4 2023-04-22 14:56:23,710 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4, entries=150, sequenceid=504, filesize=11.9 K 2023-04-22 14:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/3042a20a2ab24d13820e776861ca0435 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435 2023-04-22 14:56:23,714 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435, entries=150, sequenceid=504, filesize=11.9 K 2023-04-22 14:56:23,715 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 89f2897412e14388617caefccca798eb in 453ms, sequenceid=504, compaction requested=true 2023-04-22 14:56:23,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:23,715 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:23,716 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:23,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b4aac0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:23,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:23,718 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,722 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:23,723 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,723 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:23,729 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:23,732 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:23,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:23,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:23,732 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:23,732 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:23,733 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,733 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,733 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,734 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,735 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,738 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:23,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,738 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,738 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:23,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:23,738 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:23,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-22 14:56:23,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:23,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:23,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:23,761 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042291afed5bebfa4ec18a0b7198c8396ef4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042291afed5bebfa4ec18a0b7198c8396ef4 2023-04-22 14:56:23,765 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/fa0fb0346d7542f49fdaccd96aaadfd0 2023-04-22 14:56:23,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175443781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175443784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,839 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:23,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:23,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:23,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:23,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:23,842 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:23,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:23,842 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:23,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,843 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,844 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,844 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:23,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:23,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:23,849 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e836f8[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:23,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:23,851 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,853 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:23,853 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,853 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:23,854 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:23,854 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,855 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:23,857 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:23,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:23,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:23,857 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:23,857 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:23,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,859 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,860 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,861 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,866 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:23,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,866 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:23,866 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:23,866 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:23,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175443885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175443888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,966 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:23,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:23,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:23,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:23,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:23,970 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:23,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:23,970 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:23,971 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,971 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,972 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,972 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,973 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:23,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:23,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:23,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:23,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:23,974 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:23,976 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:23,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de32fe3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:23,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:23,977 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,979 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:23,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,979 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:23,986 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:23,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:23,989 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:23,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:23,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:23,989 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:23,989 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:23,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:23,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,991 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,993 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:23,995 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:23,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:23,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:23,995 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:23,995 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:23,995 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:23,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:24,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175444089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175444093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,096 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:24,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:24,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:24,100 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:24,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:24,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:24,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:24,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,100 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,101 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,101 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:24,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,102 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:24,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:24,103 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,104 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:24,104 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:24,104 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:24,106 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:24,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,107 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76054be6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:24,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:24,109 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,110 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:24,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,110 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:24,112 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:24,112 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:24,115 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:24,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:24,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:24,115 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:24,115 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:24,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:24,125 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:24,126 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175444124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175444124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,129 DEBUG [Thread-3733] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:24,130 DEBUG [Thread-3737] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:24,130 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:24,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:24,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:24,132 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:24,133 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:24,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:24,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:24,133 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:24,133 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:24,133 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:24,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:24,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175444137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,140 DEBUG [Thread-3729] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:24,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/2dcba5055adf492f8833b84738a1393a 2023-04-22 14:56:24,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/32d179995d924e7abf11627d3102d805 2023-04-22 14:56:24,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/fa0fb0346d7542f49fdaccd96aaadfd0 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0 2023-04-22 14:56:24,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0, entries=250, sequenceid=517, filesize=39.2 K 2023-04-22 14:56:24,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/2dcba5055adf492f8833b84738a1393a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a 2023-04-22 14:56:24,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 14:56:24,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/32d179995d924e7abf11627d3102d805 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805 2023-04-22 14:56:24,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805, entries=150, sequenceid=517, filesize=11.9 K 2023-04-22 14:56:24,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 89f2897412e14388617caefccca798eb in 458ms, sequenceid=517, compaction requested=true 2023-04-22 14:56:24,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:24,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:24,202 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:56:24,202 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:56:24,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:24,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:24,204 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:56:24,204 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:24,204 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,204 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e97946074467444389042ab53c680c6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=60.1 K 2023-04-22 14:56:24,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 196910 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 14:56:24,204 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:24,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,204 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f6afd0f2337b45db9ba2b5198a6ebc89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=192.3 K 2023-04-22 14:56:24,204 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting e97946074467444389042ab53c680c6a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1682175379934 2023-04-22 14:56:24,205 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting f6afd0f2337b45db9ba2b5198a6ebc89, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682175377815 2023-04-22 14:56:24,205 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c46a524bc9904a8c859050793b2c3c5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682175379986 2023-04-22 14:56:24,205 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting d5686fc30b9d491db602a825528c4af1, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682175378922 2023-04-22 14:56:24,205 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3f7f40ebf1aa47fead6bc199dd946d73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1682175380591 2023-04-22 14:56:24,205 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 9a9510dec03542dd95c8c09ee5d67d52, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1682175379934 2023-04-22 14:56:24,206 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 8a6e5293c2d745edbadbf852ab9d5fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682175382793 2023-04-22 14:56:24,206 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3362d16bbe554efbb50eeab6891f36ca, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682175379986 2023-04-22 14:56:24,206 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2dcba5055adf492f8833b84738a1393a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383436 2023-04-22 14:56:24,206 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting aef5f1757ff3428393fe1ee3842d22e6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1682175380591 2023-04-22 14:56:24,206 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 4100718db34d481e94f4ac0bb7c7b760, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682175382793 2023-04-22 14:56:24,207 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting fa0fb0346d7542f49fdaccd96aaadfd0, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383427 2023-04-22 14:56:24,220 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#577 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:24,226 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#578 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:24,233 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:24,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:24,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:24,244 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:24,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:24,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:24,244 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:24,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,245 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,246 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:24,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:24,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,246 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:24,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:24,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:24,247 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:24,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:24,251 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,251 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:24,251 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:24,252 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:24,252 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:24,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:24,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175444442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175444447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175444548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175444553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,650 WARN [DataStreamer for file /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/413ab3b5ade549b580414c2c46ef1457] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-22 14:56:24,655 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/9b888bb9a6704120a43747eea086a7d1 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9b888bb9a6704120a43747eea086a7d1 2023-04-22 14:56:24,657 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/413ab3b5ade549b580414c2c46ef1457 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/413ab3b5ade549b580414c2c46ef1457 2023-04-22 14:56:24,660 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into 9b888bb9a6704120a43747eea086a7d1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:24,660 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:24,660 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=11, startTime=1682175384202; duration=0sec 2023-04-22 14:56:24,660 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:24,660 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:56:24,662 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:56:24,662 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:24,662 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:24,662 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0abd5e8a88114e03806ad6a4914444eb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=60.1 K 2023-04-22 14:56:24,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0abd5e8a88114e03806ad6a4914444eb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1682175379934 2023-04-22 14:56:24,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting d2b97b61b3f4486180b4cd7569e220e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682175379986 2023-04-22 14:56:24,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting b7699effca544ea2bc9742d6293aec08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1682175380591 2023-04-22 14:56:24,664 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 3042a20a2ab24d13820e776861ca0435, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1682175382793 2023-04-22 14:56:24,664 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into 413ab3b5ade549b580414c2c46ef1457(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:24,664 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:24,664 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=9, startTime=1682175384202; duration=0sec 2023-04-22 14:56:24,664 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 32d179995d924e7abf11627d3102d805, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383436 2023-04-22 14:56:24,664 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:24,670 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422035166b0ff844a26839eebe334686060 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422035166b0ff844a26839eebe334686060 2023-04-22 14:56:24,677 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:24,700 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/e361e36b2e594508aedf2b1f8b194d1d 2023-04-22 14:56:24,708 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/0db23fee95f64b3b80f5d883410f9953 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0db23fee95f64b3b80f5d883410f9953 2023-04-22 14:56:24,713 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into 0db23fee95f64b3b80f5d883410f9953(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:24,713 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:24,713 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=11, startTime=1682175384202; duration=0sec 2023-04-22 14:56:24,713 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:24,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175444753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:24,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:24,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175444755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175445055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175445058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,111 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/6c5df576164149e88f778258c369414d 2023-04-22 14:56:25,120 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/56b43699fb2542aea729ecb1afe25a14 2023-04-22 14:56:25,124 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/e361e36b2e594508aedf2b1f8b194d1d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d 2023-04-22 14:56:25,128 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d, entries=150, sequenceid=536, filesize=25.4 K 2023-04-22 14:56:25,129 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/6c5df576164149e88f778258c369414d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d 2023-04-22 14:56:25,134 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d, entries=150, sequenceid=536, filesize=11.9 K 2023-04-22 14:56:25,134 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/56b43699fb2542aea729ecb1afe25a14 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14 2023-04-22 14:56:25,138 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14, entries=150, sequenceid=536, filesize=11.9 K 2023-04-22 14:56:25,138 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 89f2897412e14388617caefccca798eb in 886ms, sequenceid=536, compaction requested=false 2023-04-22 14:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,140 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13820d4c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,142 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,142 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,144 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,144 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,146 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,146 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,148 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,148 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,150 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,150 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,151 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:25,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,154 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,155 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,155 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,255 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:25,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:25,258 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:25,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:25,258 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:25,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,259 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,264 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,264 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,265 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:25,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:25,266 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,266 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:25,267 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,285 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304223b617084f0694ff7872a36b94b72032a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223b617084f0694ff7872a36b94b72032a 2023-04-22 14:56:25,290 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=552, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/0942dc7e93b84db69ac550483af9ffdc 2023-04-22 14:56:25,302 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/73da7cdfd2dc4fdf96d709205f5134a4 2023-04-22 14:56:25,343 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/2309a1664cb6495bb615d0391c630dca 2023-04-22 14:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/0942dc7e93b84db69ac550483af9ffdc as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc 2023-04-22 14:56:25,351 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc, entries=150, sequenceid=552, filesize=25.4 K 2023-04-22 14:56:25,351 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/73da7cdfd2dc4fdf96d709205f5134a4 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4 2023-04-22 14:56:25,355 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4, entries=150, sequenceid=552, filesize=11.9 K 2023-04-22 14:56:25,355 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/2309a1664cb6495bb615d0391c630dca as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca 2023-04-22 14:56:25,359 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca, entries=150, sequenceid=552, filesize=11.9 K 2023-04-22 14:56:25,360 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 89f2897412e14388617caefccca798eb in 93ms, sequenceid=552, compaction requested=true 2023-04-22 14:56:25,360 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,360 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,362 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,362 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f8a632[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,366 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,367 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,367 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,368 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,368 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,371 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,371 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,371 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,374 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,374 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,374 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,375 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,376 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,379 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,379 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,379 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,379 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,479 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:25,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:25,483 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,484 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,485 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,485 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:25,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,489 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2500b39[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,491 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,493 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,494 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,495 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,496 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,498 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,499 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,499 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,500 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,501 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,502 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,508 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,508 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,509 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:25,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,509 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,509 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:25,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-22 14:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:25,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:25,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:25,608 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422e2bbe1b1614a47c29413d7030cb86894 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2bbe1b1614a47c29413d7030cb86894 2023-04-22 14:56:25,609 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:25,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:25,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:25,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:25,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:25,613 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:25,613 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,613 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:25,613 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:25,614 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=563, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c5553936195b4372943d50d1acb0f089 2023-04-22 14:56:25,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,617 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,618 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,619 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,619 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:25,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:25,621 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,622 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e562aee[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,625 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,626 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,627 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,628 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/89ed0b21547f4b8a942993a37fac1c6a 2023-04-22 14:56:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,633 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,634 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,637 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,637 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,637 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,639 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,640 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,642 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/7c04a8650a9b45598f66b3a3a5fc5090 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,645 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,645 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,645 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,645 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c5553936195b4372943d50d1acb0f089 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089 2023-04-22 14:56:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682175385638, max timestamp=1682175385647Num uniques 859; 2023-04-22 14:56:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682175385638, max timestamp=1682175385647Num uniques 859; 2023-04-22 14:56:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682175385638, max timestamp=1682175385647Num uniques 859; 2023-04-22 14:56:25,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-22 14:56:25,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089, entries=300, sequenceid=563, filesize=46.1 K 2023-04-22 14:56:25,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/89ed0b21547f4b8a942993a37fac1c6a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a 2023-04-22 14:56:25,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a, entries=100, sequenceid=563, filesize=9.5 K 2023-04-22 14:56:25,655 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-22 14:56:25,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/7c04a8650a9b45598f66b3a3a5fc5090 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090 2023-04-22 14:56:25,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090, entries=100, sequenceid=563, filesize=9.5 K 2023-04-22 14:56:25,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 89f2897412e14388617caefccca798eb in 63ms, sequenceid=563, compaction requested=true 2023-04-22 14:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:25,660 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:25,660 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:25,661 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 126054 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:25,661 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:25,661 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,661 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/413ab3b5ade549b580414c2c46ef1457, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=123.1 K 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 413ab3b5ade549b580414c2c46ef1457, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383436 2023-04-22 14:56:25,662 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,662 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9b888bb9a6704120a43747eea086a7d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=46.0 K 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting e361e36b2e594508aedf2b1f8b194d1d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682175383767 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 9b888bb9a6704120a43747eea086a7d1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383436 2023-04-22 14:56:25,662 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 0942dc7e93b84db69ac550483af9ffdc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682175384435 2023-04-22 14:56:25,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6c5df576164149e88f778258c369414d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682175383767 2023-04-22 14:56:25,663 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c5553936195b4372943d50d1acb0f089, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175385579 2023-04-22 14:56:25,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 73da7cdfd2dc4fdf96d709205f5134a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682175384435 2023-04-22 14:56:25,663 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 89ed0b21547f4b8a942993a37fac1c6a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175385588 2023-04-22 14:56:25,677 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:25,680 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#590 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:25,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:25,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=73.80 KB heapSize=201 KB 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:25,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-22 14:56:25,692 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/0f5ad8ed191a4d9db21d2aa849ca2808 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0f5ad8ed191a4d9db21d2aa849ca2808 2023-04-22 14:56:25,697 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into 0f5ad8ed191a4d9db21d2aa849ca2808(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:25,698 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,698 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=12, startTime=1682175385660; duration=0sec 2023-04-22 14:56:25,698 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:25,698 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:56:25,700 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-22 14:56:25,700 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:25,700 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,700 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0db23fee95f64b3b80f5d883410f9953, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=46.0 K 2023-04-22 14:56:25,700 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0db23fee95f64b3b80f5d883410f9953, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682175383436 2023-04-22 14:56:25,701 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 56b43699fb2542aea729ecb1afe25a14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682175383767 2023-04-22 14:56:25,701 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 2309a1664cb6495bb615d0391c630dca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682175384435 2023-04-22 14:56:25,702 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7c04a8650a9b45598f66b3a3a5fc5090, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175385588 2023-04-22 14:56:25,717 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#592 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:25,721 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042227205013a8834411bfc0448e71959e57 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042227205013a8834411bfc0448e71959e57 2023-04-22 14:56:25,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175445731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175445732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,745 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:25,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:25,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:25,761 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/32064d4528c141a6898c53d338cf8f15 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/32064d4528c141a6898c53d338cf8f15 2023-04-22 14:56:25,765 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into 32064d4528c141a6898c53d338cf8f15(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:25,765 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:25,765 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=12, startTime=1682175385660; duration=0sec 2023-04-22 14:56:25,765 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:25,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:25,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:25,767 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:25,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:25,767 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:25,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,768 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,769 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=590, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3618ae5a349948b5a033e43782562e12 2023-04-22 14:56:25,769 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,769 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,770 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:25,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:25,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:25,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:25,772 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,775 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbda405[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,777 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,777 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,778 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,778 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,779 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,782 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,782 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,782 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,783 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,784 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,784 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,784 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,785 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,789 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,789 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,789 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,789 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175445834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175445834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,889 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:25,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:25,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:25,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:25,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:25,893 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:25,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:25,893 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:25,894 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,894 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,895 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,895 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:25,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:25,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:25,897 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,897 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:25,897 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:25,897 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:25,898 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,898 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,899 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,899 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8750b6[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:25,900 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:25,900 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,902 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:25,902 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,902 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,904 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:25,907 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:25,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:25,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:25,907 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:25,907 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:25,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,908 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:25,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:25,910 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:25,911 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,915 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:25,915 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:25,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:25,915 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:25,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:25,915 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:25,916 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:25,916 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:25,915 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:25,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:25,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:25,916 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,016 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:26,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:26,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:26,019 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:26,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:26,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:26,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,020 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,022 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:26,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:26,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,022 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:26,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:26,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:26,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:26,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:26,024 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:26,026 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,026 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,027 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:26,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0c8f9c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:26,028 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:26,028 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,029 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,029 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:26,031 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:26,034 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:26,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:26,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:26,034 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:26,034 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:26,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:26,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:26,037 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,038 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,039 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:26,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:26,042 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175446039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:26,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175446039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:26,042 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:26,042 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:26,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,042 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,042 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,042 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,042 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,041 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:26,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:26,142 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:26,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:26,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:26,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:26,146 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:26,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:26,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:26,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,146 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,148 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:26,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:26,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,148 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:26,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:26,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:26,149 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:26,150 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:26,150 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:26,150 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:26,152 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e2d1a9[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:26,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:26,154 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,156 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:26,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,157 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:26,158 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:26,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:26,158 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:26,159 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/c27c76376d44407ba9f46a6688aa8ab7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/c27c76376d44407ba9f46a6688aa8ab7 2023-04-22 14:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:26,163 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:26,163 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:26,163 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:26,163 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:26,163 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:26,164 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into c27c76376d44407ba9f46a6688aa8ab7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:26,165 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:26,165 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=12, startTime=1682175385660; duration=0sec 2023-04-22 14:56:26,165 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:26,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,165 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:26,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,168 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,169 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:26,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:26,171 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,172 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:26,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:26,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:26,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:26,172 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,172 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:26,172 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:26,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:26,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/da149a2676794da7bd4b03ebbc0fb12a 2023-04-22 14:56:26,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/90fd8e1d5cd84bb3ae65a0ed544cb31f 2023-04-22 14:56:26,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/3618ae5a349948b5a033e43782562e12 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12 2023-04-22 14:56:26,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12, entries=150, sequenceid=590, filesize=25.4 K 2023-04-22 14:56:26,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/da149a2676794da7bd4b03ebbc0fb12a as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a 2023-04-22 14:56:26,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a, entries=150, sequenceid=590, filesize=11.9 K 2023-04-22 14:56:26,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/90fd8e1d5cd84bb3ae65a0ed544cb31f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f 2023-04-22 14:56:26,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f, entries=150, sequenceid=590, filesize=11.9 K 2023-04-22 14:56:26,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~200.95 KB/205776, currentSize=107.34 KB/109920 for 89f2897412e14388617caefccca798eb in 545ms, sequenceid=590, compaction requested=false 2023-04-22 14:56:26,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:26,272 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:26,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:26,277 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:26,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:26,277 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:26,278 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,278 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,279 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:26,279 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:26,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,280 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:26,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:26,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:26,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:26,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:26,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:26,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:26,282 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:26,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:26,282 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:26,282 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:26,283 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:26,300 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422515d083be6e9453a93b1900f473891aa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422515d083be6e9453a93b1900f473891aa 2023-04-22 14:56:26,310 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=612, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/97b72bfd592f43979d24801cf290304e 2023-04-22 14:56:26,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:26,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:26,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175446374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175446374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175446477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175446478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175446681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175446681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,721 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/fe4e2e15894148d5b97b517b6755f822 2023-04-22 14:56:26,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175446983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:26,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:26,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175446984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,131 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/7a00edeb4ae24d22ac1406c52d5adee2 2023-04-22 14:56:27,136 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/97b72bfd592f43979d24801cf290304e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e 2023-04-22 14:56:27,140 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e, entries=150, sequenceid=612, filesize=25.4 K 2023-04-22 14:56:27,140 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/fe4e2e15894148d5b97b517b6755f822 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822 2023-04-22 14:56:27,144 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822, entries=150, sequenceid=612, filesize=11.9 K 2023-04-22 14:56:27,145 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/7a00edeb4ae24d22ac1406c52d5adee2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2 2023-04-22 14:56:27,148 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2, entries=150, sequenceid=612, filesize=11.9 K 2023-04-22 14:56:27,149 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 89f2897412e14388617caefccca798eb in 867ms, sequenceid=612, compaction requested=true 2023-04-22 14:56:27,149 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:27,149 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,151 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cfd85c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:27,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:27,153 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,155 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:27,155 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,156 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:27,157 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:27,159 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:27,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:27,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:27,160 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:27,160 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:27,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:27,161 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:27,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:27,162 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,163 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:27,166 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:27,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:27,166 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:27,166 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:27,166 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:27,267 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:27,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:27,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:56:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-22 14:56:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-22 14:56:27,270 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-22 14:56:27,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:27,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-22 14:56:27,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-22 14:56:27,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,270 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:27,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,273 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:27,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:27,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,273 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:27,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:27,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-22 14:56:27,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-22 14:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-22 14:56:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-22 14:56:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:56:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-22 14:56:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:56:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:56:27,274 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. started... 2023-04-22 14:56:27,275 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:27,275 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:27,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. as already flushing 2023-04-22 14:56:27,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175447507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175447509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175447610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175447612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,693 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304227f93064e162b448fb2df2e1c193002f7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f93064e162b448fb2df2e1c193002f7 2023-04-22 14:56:27,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175447813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:27,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175447815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,099 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=626, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c8d063962d634cbdb4e5b8429a02bb30 2023-04-22 14:56:28,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175448118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175448119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44902 deadline: 1682175448150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,156 DEBUG [Thread-3733] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:28,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32935] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44942 deadline: 1682175448154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,159 DEBUG [Thread-3737] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44956 deadline: 1682175448156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,161 DEBUG [Thread-3729] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., hostname=jenkins-hbase4.apache.org,32935,1682175218140, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:56:28,510 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/688a0733ad5d4617a14117027ce4f7dd 2023-04-22 14:56:28,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44908 deadline: 1682175448624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-22 14:56:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=32935] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44928 deadline: 1682175448624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,920 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/41986e181f3e43d2be4aff74beb314d2 2023-04-22 14:56:28,924 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/c8d063962d634cbdb4e5b8429a02bb30 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30 2023-04-22 14:56:28,927 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30, entries=150, sequenceid=626, filesize=25.4 K 2023-04-22 14:56:28,928 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/688a0733ad5d4617a14117027ce4f7dd as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd 2023-04-22 14:56:28,941 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd, entries=150, sequenceid=626, filesize=11.9 K 2023-04-22 14:56:28,942 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/41986e181f3e43d2be4aff74beb314d2 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2 2023-04-22 14:56:28,945 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2, entries=150, sequenceid=626, filesize=11.9 K 2023-04-22 14:56:28,946 INFO [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 89f2897412e14388617caefccca798eb in 1671ms, sequenceid=626, compaction requested=true 2023-04-22 14:56:28,946 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:28,946 DEBUG [rs(jenkins-hbase4.apache.org,32935,1682175218140)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-22 14:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-22 14:56:28,953 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:28,954 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,32935,1682175218140' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-22 14:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67593473[Count = 0] remaining members to acquire global barrier 2023-04-22 14:56:28,955 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-22 14:56:28,955 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,956 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-22 14:56:28,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-22 14:56:28,957 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,32935,1682175218140' in zk 2023-04-22 14:56:28,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,957 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:56:28,960 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-22 14:56:28,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:28,960 DEBUG [member: 'jenkins-hbase4.apache.org,32935,1682175218140' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-22 14:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,32935,1682175218140': 2023-04-22 14:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,32935,1682175218140' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-22 14:56:28,963 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-22 14:56:28,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:56:28,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:56:28,963 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-22 14:56:28,963 INFO [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:56:28,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:28,964 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:56:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:28,965 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:56:28,966 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-22 14:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-22 14:56:28,968 DEBUG [(jenkins-hbase4.apache.org,37547,1682175216913)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:56:28,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-22 14:56:28,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:56:28,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:56:28,968 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:56:28,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-22 14:56:28,969 DEBUG [Thread-3739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-22 14:56:28,969 DEBUG [Thread-3739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:56:28,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-22 14:56:28,969 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-22 14:56:28,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:56:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:56:29,012 DEBUG [Thread-3742] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:51853 2023-04-22 14:56:29,013 DEBUG [Thread-3742] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,013 DEBUG [Thread-3746] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:51853 2023-04-22 14:56:29,013 DEBUG [Thread-3746] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,014 DEBUG [Thread-3740] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:51853 2023-04-22 14:56:29,014 DEBUG [Thread-3740] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,015 DEBUG [Thread-3748] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:51853 2023-04-22 14:56:29,015 DEBUG [Thread-3748] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,016 DEBUG [Thread-3744] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:51853 2023-04-22 14:56:29,016 DEBUG [Thread-3744] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,069 DEBUG [Thread-3739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:56:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:56:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=32935] regionserver.HRegion(9158): Flush requested on 89f2897412e14388617caefccca798eb 2023-04-22 14:56:29,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-22 14:56:29,636 DEBUG [Thread-3735] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:51853 2023-04-22 14:56:29,636 DEBUG [Thread-3735] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:29,637 DEBUG [Thread-3731] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:51853 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:29,637 DEBUG [Thread-3731] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:29,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:30,048 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422d17695d93b684dfab62d49c3632a2042 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d17695d93b684dfab62d49c3632a2042 2023-04-22 14:56:30,453 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=646, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/33186e7cbd374362af69024f9700669f 2023-04-22 14:56:30,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c66a6783593c417abef849565b5603d9 2023-04-22 14:56:31,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/6e9b0a3547504fc6b024e434dd2087f7 2023-04-22 14:56:31,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/33186e7cbd374362af69024f9700669f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f 2023-04-22 14:56:31,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f, entries=150, sequenceid=646, filesize=25.4 K 2023-04-22 14:56:31,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/c66a6783593c417abef849565b5603d9 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9 2023-04-22 14:56:31,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9, entries=150, sequenceid=646, filesize=11.9 K 2023-04-22 14:56:31,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/6e9b0a3547504fc6b024e434dd2087f7 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7 2023-04-22 14:56:31,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7, entries=150, sequenceid=646, filesize=11.9 K 2023-04-22 14:56:31,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 89f2897412e14388617caefccca798eb in 1647ms, sequenceid=646, compaction requested=true 2023-04-22 14:56:31,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:31,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:31,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:31,283 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:56:31,283 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:56:31,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 131116 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/A is initiating minor compaction (all files) 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/B is initiating minor compaction (all files) 2023-04-22 14:56:31,285 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/A in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:31,285 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/B in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:31,285 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/32064d4528c141a6898c53d338cf8f15, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=128.0 K 2023-04-22 14:56:31,285 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0f5ad8ed191a4d9db21d2aa849ca2808, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=60.4 K 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 32064d4528c141a6898c53d338cf8f15, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175384446 2023-04-22 14:56:31,285 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 0f5ad8ed191a4d9db21d2aa849ca2808, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175384446 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 3618ae5a349948b5a033e43782562e12, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682175385652 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting da149a2676794da7bd4b03ebbc0fb12a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682175385652 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 97b72bfd592f43979d24801cf290304e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1682175385719 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting fe4e2e15894148d5b97b517b6755f822, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1682175385719 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting c8d063962d634cbdb4e5b8429a02bb30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682175386349 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 688a0733ad5d4617a14117027ce4f7dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682175386349 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] compactions.Compactor(207): Compacting 33186e7cbd374362af69024f9700669f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1682175387508 2023-04-22 14:56:31,286 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c66a6783593c417abef849565b5603d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1682175387508 2023-04-22 14:56:31,294 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#B#compaction#604 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:31,296 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#A#compaction#605 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:31,701 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/f1db94c379ac490da9ce12fd9f8a123d as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/f1db94c379ac490da9ce12fd9f8a123d 2023-04-22 14:56:31,705 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 89f2897412e14388617caefccca798eb/B of 89f2897412e14388617caefccca798eb into f1db94c379ac490da9ce12fd9f8a123d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:31,705 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:31,705 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/B, priority=11, startTime=1682175391283; duration=0sec 2023-04-22 14:56:31,705 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:56:31,705 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-22 14:56:31,706 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-22 14:56:31,706 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1912): 89f2897412e14388617caefccca798eb/C is initiating minor compaction (all files) 2023-04-22 14:56:31,706 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 89f2897412e14388617caefccca798eb/C in TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:31,706 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/c27c76376d44407ba9f46a6688aa8ab7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7] into tmpdir=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp, totalSize=60.4 K 2023-04-22 14:56:31,707 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting c27c76376d44407ba9f46a6688aa8ab7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682175384446 2023-04-22 14:56:31,707 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 90fd8e1d5cd84bb3ae65a0ed544cb31f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682175385652 2023-04-22 14:56:31,707 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 7a00edeb4ae24d22ac1406c52d5adee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1682175385719 2023-04-22 14:56:31,707 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 41986e181f3e43d2be4aff74beb314d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682175386349 2023-04-22 14:56:31,708 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] compactions.Compactor(207): Compacting 6e9b0a3547504fc6b024e434dd2087f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1682175387508 2023-04-22 14:56:31,714 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] throttle.PressureAwareThroughputController(145): 89f2897412e14388617caefccca798eb#C#compaction#606 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:56:32,121 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/f156643cd8c0482183a513e1c89a1340 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f156643cd8c0482183a513e1c89a1340 2023-04-22 14:56:32,124 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 89f2897412e14388617caefccca798eb/C of 89f2897412e14388617caefccca798eb into f156643cd8c0482183a513e1c89a1340(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:56:32,125 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:32,125 INFO [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/C, priority=11, startTime=1682175391283; duration=0sec 2023-04-22 14:56:32,125 DEBUG [RS:0;jenkins-hbase4:32935-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:32,511 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/f560ef77747644248bef364f87d3949f as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f560ef77747644248bef364f87d3949f 2023-04-22 14:56:32,515 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 89f2897412e14388617caefccca798eb/A of 89f2897412e14388617caefccca798eb into f560ef77747644248bef364f87d3949f(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-22 14:56:32,515 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:32,515 INFO [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb., storeName=89f2897412e14388617caefccca798eb/A, priority=11, startTime=1682175391283; duration=1sec 2023-04-22 14:56:32,515 DEBUG [RS:0;jenkins-hbase4:32935-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:56:38,184 DEBUG [Thread-3737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:51853 2023-04-22 14:56:38,184 DEBUG [Thread-3737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:38,239 DEBUG [Thread-3733] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:51853 2023-04-22 14:56:38,239 DEBUG [Thread-3733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:38,243 DEBUG [Thread-3729] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:51853 2023-04-22 14:56:38,243 DEBUG [Thread-3729] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:38,243 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-22 14:56:38,243 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-22 14:56:38,243 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 4498 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 4243 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 4426 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 4473 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(398): read 4236 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-22 14:56:38,244 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:56:38,244 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:51853 2023-04-22 14:56:38,244 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:38,245 INFO [Listener at localhost/45141] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-22 14:56:38,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-22 14:56:38,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:38,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:38,248 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175398248"}]},"ts":"1682175398248"} 2023-04-22 14:56:38,248 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-22 14:56:38,250 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-22 14:56:38,250 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, UNASSIGN}] 2023-04-22 14:56:38,251 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, UNASSIGN 2023-04-22 14:56:38,252 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:38,252 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175398252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175398252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175398252"}]},"ts":"1682175398252"} 2023-04-22 14:56:38,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140}] 2023-04-22 14:56:38,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:38,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 89f2897412e14388617caefccca798eb 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 89f2897412e14388617caefccca798eb, disabling compactions & flushes 2023-04-22 14:56:38,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. after waiting 0 ms 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:38,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 89f2897412e14388617caefccca798eb 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=A 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=B 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 89f2897412e14388617caefccca798eb, store=C 2023-04-22 14:56:38,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-22 14:56:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:38,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230422a0b151c0cd6740c59f51f70ed700cff8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a0b151c0cd6740c59f51f70ed700cff8 2023-04-22 14:56:38,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:39,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=655, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/9b2f3885046e4a39be460b6fdf460526 2023-04-22 14:56:39,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:39,529 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:56:39,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/726c1db47ea84a3ea0366da7256fc021 2023-04-22 14:56:40,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/d4d9ff47c654432dbc3236438a90f236 2023-04-22 14:56:40,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/A/9b2f3885046e4a39be460b6fdf460526 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9b2f3885046e4a39be460b6fdf460526 2023-04-22 14:56:40,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9b2f3885046e4a39be460b6fdf460526, entries=50, sequenceid=655, filesize=11.7 K 2023-04-22 14:56:40,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/B/726c1db47ea84a3ea0366da7256fc021 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/726c1db47ea84a3ea0366da7256fc021 2023-04-22 14:56:40,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/726c1db47ea84a3ea0366da7256fc021, entries=50, sequenceid=655, filesize=7.1 K 2023-04-22 14:56:40,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/.tmp/C/d4d9ff47c654432dbc3236438a90f236 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d4d9ff47c654432dbc3236438a90f236 2023-04-22 14:56:40,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d4d9ff47c654432dbc3236438a90f236, entries=50, sequenceid=655, filesize=7.1 K 2023-04-22 14:56:40,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 89f2897412e14388617caefccca798eb in 1645ms, sequenceid=655, compaction requested=false 2023-04-22 14:56:40,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e97129b9115f4e638df7422a09322ec1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/55c875301ec24e4a925ce578f8d0d460, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f6afd0f2337b45db9ba2b5198a6ebc89, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/413ab3b5ade549b580414c2c46ef1457, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/32064d4528c141a6898c53d338cf8f15, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f] to archive 2023-04-22 14:56:40,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:40,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/547361f3a5674052829477f012323880 2023-04-22 14:56:40,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7623c3b769d94a0d8613ab5f35543c35 2023-04-22 14:56:40,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7121708895384e31b6f5feabe407ef80 2023-04-22 14:56:40,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e97129b9115f4e638df7422a09322ec1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e97129b9115f4e638df7422a09322ec1 2023-04-22 14:56:40,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/48461b99eeb54211a3187c12525618ee 2023-04-22 14:56:40,080 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/7a21f2b683354971bc8c1163dfadb36a 2023-04-22 14:56:40,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c2c2001c5bd1456cb9dab8868f9ce404 2023-04-22 14:56:40,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/55c875301ec24e4a925ce578f8d0d460 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/55c875301ec24e4a925ce578f8d0d460 2023-04-22 14:56:40,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3ee83f39e78e444f9e4418b49113ad96 2023-04-22 14:56:40,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3f37691557c245aea3d2d324f22302e3 2023-04-22 14:56:40,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/76209e07868c4218a04317bb1f7d5d76 2023-04-22 14:56:40,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/ab17a8db22184982b7ded27b797d0760 2023-04-22 14:56:40,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e2caf953ac224ceba487a425ed83e8f5 2023-04-22 14:56:40,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/bfbed64c287f455f8fca1e7e68f66cf3 2023-04-22 14:56:40,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3a25649eeeab4df5a7722cdeeaf4050d 2023-04-22 14:56:40,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f6afd0f2337b45db9ba2b5198a6ebc89 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f6afd0f2337b45db9ba2b5198a6ebc89 2023-04-22 14:56:40,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/d5686fc30b9d491db602a825528c4af1 2023-04-22 14:56:40,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9a9510dec03542dd95c8c09ee5d67d52 2023-04-22 14:56:40,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3362d16bbe554efbb50eeab6891f36ca 2023-04-22 14:56:40,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/aef5f1757ff3428393fe1ee3842d22e6 2023-04-22 14:56:40,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/4100718db34d481e94f4ac0bb7c7b760 2023-04-22 14:56:40,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/fa0fb0346d7542f49fdaccd96aaadfd0 2023-04-22 14:56:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/413ab3b5ade549b580414c2c46ef1457 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/413ab3b5ade549b580414c2c46ef1457 2023-04-22 14:56:40,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/e361e36b2e594508aedf2b1f8b194d1d 2023-04-22 14:56:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/0942dc7e93b84db69ac550483af9ffdc 2023-04-22 14:56:40,097 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c5553936195b4372943d50d1acb0f089 2023-04-22 14:56:40,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/32064d4528c141a6898c53d338cf8f15 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/32064d4528c141a6898c53d338cf8f15 2023-04-22 14:56:40,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/3618ae5a349948b5a033e43782562e12 2023-04-22 14:56:40,099 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/97b72bfd592f43979d24801cf290304e 2023-04-22 14:56:40,100 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/c8d063962d634cbdb4e5b8429a02bb30 2023-04-22 14:56:40,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/33186e7cbd374362af69024f9700669f 2023-04-22 14:56:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/d5fd403c78804b4e9b7897b5adbbd35f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/29d6fa2e01584cdab177d5eb59dffa05, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3b6d93ad5ee64b168d1ea85c87815134, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e97946074467444389042ab53c680c6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9b888bb9a6704120a43747eea086a7d1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0f5ad8ed191a4d9db21d2aa849ca2808, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9] to archive 2023-04-22 14:56:40,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:40,106 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0fd69fb02f8f41a2945f643cb6439cda 2023-04-22 14:56:40,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a4b49550a4fe46a2bba0d79058a8af10 2023-04-22 14:56:40,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/d5fd403c78804b4e9b7897b5adbbd35f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/d5fd403c78804b4e9b7897b5adbbd35f 2023-04-22 14:56:40,108 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/be8522042e7c4dfab8f85a169befca34 2023-04-22 14:56:40,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9aacb413678d4106b432f407cca065d3 2023-04-22 14:56:40,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2cd633a1d1fd42c1b3c025d80c977f6d 2023-04-22 14:56:40,111 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/29d6fa2e01584cdab177d5eb59dffa05 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/29d6fa2e01584cdab177d5eb59dffa05 2023-04-22 14:56:40,112 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/a0b2e599e8f449d690cbd73314afc9d9 2023-04-22 14:56:40,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/63cf96732b874847b6f9ec500b74547e 2023-04-22 14:56:40,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/4532cf61a85b4b1fa6766baf97007c18 2023-04-22 14:56:40,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c3cb80a4827147ee9736b38139b67c10 2023-04-22 14:56:40,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/86743c5ead3a43aeb836581096474f75 2023-04-22 14:56:40,116 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e67af1b17dcb4f84a7e70c6d6898db74 2023-04-22 14:56:40,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/81a1ce8e0e8d49c3ab9ffbb5d3ba92e0 2023-04-22 14:56:40,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3b6d93ad5ee64b168d1ea85c87815134 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3b6d93ad5ee64b168d1ea85c87815134 2023-04-22 14:56:40,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0be616ab79e849f8abe08195fbaef3f2 2023-04-22 14:56:40,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/748640e8ab484947b3e7cfe0e77230f7 2023-04-22 14:56:40,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e97946074467444389042ab53c680c6a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/e97946074467444389042ab53c680c6a 2023-04-22 14:56:40,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/56298b24f7a94ddc9c83db50d9c3c5a6 2023-04-22 14:56:40,122 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c46a524bc9904a8c859050793b2c3c5c 2023-04-22 14:56:40,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/3f7f40ebf1aa47fead6bc199dd946d73 2023-04-22 14:56:40,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/8a6e5293c2d745edbadbf852ab9d5fd4 2023-04-22 14:56:40,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9b888bb9a6704120a43747eea086a7d1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/9b888bb9a6704120a43747eea086a7d1 2023-04-22 14:56:40,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/2dcba5055adf492f8833b84738a1393a 2023-04-22 14:56:40,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/6c5df576164149e88f778258c369414d 2023-04-22 14:56:40,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/73da7cdfd2dc4fdf96d709205f5134a4 2023-04-22 14:56:40,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0f5ad8ed191a4d9db21d2aa849ca2808 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/0f5ad8ed191a4d9db21d2aa849ca2808 2023-04-22 14:56:40,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/89ed0b21547f4b8a942993a37fac1c6a 2023-04-22 14:56:40,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/da149a2676794da7bd4b03ebbc0fb12a 2023-04-22 14:56:40,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/fe4e2e15894148d5b97b517b6755f822 2023-04-22 14:56:40,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/688a0733ad5d4617a14117027ce4f7dd 2023-04-22 14:56:40,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/c66a6783593c417abef849565b5603d9 2023-04-22 14:56:40,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/4179322398a94c8ba3438ca4425c227d, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c6f93bf095e43fea35bdfa7c6888504, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/790ba520ce0d4a35b68600fc05d75708, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0abd5e8a88114e03806ad6a4914444eb, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0db23fee95f64b3b80f5d883410f9953, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/c27c76376d44407ba9f46a6688aa8ab7, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7] to archive 2023-04-22 14:56:40,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:56:40,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c84140fd88c40199fd7808b4a239b48 2023-04-22 14:56:40,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3a17af15104441c09cc16ccaa7c4135d 2023-04-22 14:56:40,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/4179322398a94c8ba3438ca4425c227d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/4179322398a94c8ba3438ca4425c227d 2023-04-22 14:56:40,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e119dee04f114e9f9f8184387e1829e1 2023-04-22 14:56:40,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/9bc137656bb7465bae1e94ede6764f5f 2023-04-22 14:56:40,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/bcf2be7b582b41aa964f800dcbfe33ed 2023-04-22 14:56:40,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c6f93bf095e43fea35bdfa7c6888504 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2c6f93bf095e43fea35bdfa7c6888504 2023-04-22 14:56:40,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3f0bdae1f5f749c0a09fe33ddae184e0 2023-04-22 14:56:40,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/497329bd39ce4957a99720fbb637cd9a 2023-04-22 14:56:40,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/1f830b28e35245f7830a721668fe7423 2023-04-22 14:56:40,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/71d758fe948c4a87a2909498b94108ff 2023-04-22 14:56:40,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/38327b981af84c4f991708525181553e 2023-04-22 14:56:40,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f40a64e7acaf4ccd8ebd9821e58127ef 2023-04-22 14:56:40,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/73f1f69ec7e7422793020f126631ac99 2023-04-22 14:56:40,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/790ba520ce0d4a35b68600fc05d75708 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/790ba520ce0d4a35b68600fc05d75708 2023-04-22 14:56:40,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/09cd9c80c1d24744b348af99680817aa 2023-04-22 14:56:40,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/e38b222bc5d2416ea11f24f46bca14dd 2023-04-22 14:56:40,150 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0abd5e8a88114e03806ad6a4914444eb to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0abd5e8a88114e03806ad6a4914444eb 2023-04-22 14:56:40,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/772d940a85ba483ab9bb103009330091 2023-04-22 14:56:40,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d2b97b61b3f4486180b4cd7569e220e0 2023-04-22 14:56:40,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/b7699effca544ea2bc9742d6293aec08 2023-04-22 14:56:40,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/3042a20a2ab24d13820e776861ca0435 2023-04-22 14:56:40,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0db23fee95f64b3b80f5d883410f9953 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/0db23fee95f64b3b80f5d883410f9953 2023-04-22 14:56:40,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/32d179995d924e7abf11627d3102d805 2023-04-22 14:56:40,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/56b43699fb2542aea729ecb1afe25a14 2023-04-22 14:56:40,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/2309a1664cb6495bb615d0391c630dca 2023-04-22 14:56:40,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/c27c76376d44407ba9f46a6688aa8ab7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/c27c76376d44407ba9f46a6688aa8ab7 2023-04-22 14:56:40,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7c04a8650a9b45598f66b3a3a5fc5090 2023-04-22 14:56:40,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/90fd8e1d5cd84bb3ae65a0ed544cb31f 2023-04-22 14:56:40,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/7a00edeb4ae24d22ac1406c52d5adee2 2023-04-22 14:56:40,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/41986e181f3e43d2be4aff74beb314d2 2023-04-22 14:56:40,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/6e9b0a3547504fc6b024e434dd2087f7 2023-04-22 14:56:40,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits/658.seqid, newMaxSeqId=658, maxSeqId=4 2023-04-22 14:56:40,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb. 2023-04-22 14:56:40,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 89f2897412e14388617caefccca798eb: 2023-04-22 14:56:40,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 89f2897412e14388617caefccca798eb 2023-04-22 14:56:40,168 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=89f2897412e14388617caefccca798eb, regionState=CLOSED 2023-04-22 14:56:40,168 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682175400168"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175400168"}]},"ts":"1682175400168"} 2023-04-22 14:56:40,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-22 14:56:40,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 89f2897412e14388617caefccca798eb, server=jenkins-hbase4.apache.org,32935,1682175218140 in 1.9160 sec 2023-04-22 14:56:40,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-22 14:56:40,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=89f2897412e14388617caefccca798eb, UNASSIGN in 1.9200 sec 2023-04-22 14:56:40,172 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175400171"}]},"ts":"1682175400171"} 2023-04-22 14:56:40,172 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-22 14:56:40,174 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-22 14:56:40,175 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9290 sec 2023-04-22 14:56:40,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-22 14:56:40,352 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-22 14:56:40,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-22 14:56:40,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-22 14:56:40,359 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:40,361 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C, FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits] 2023-04-22 14:56:40,363 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9b2f3885046e4a39be460b6fdf460526 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/9b2f3885046e4a39be460b6fdf460526 2023-04-22 14:56:40,364 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f560ef77747644248bef364f87d3949f to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/A/f560ef77747644248bef364f87d3949f 2023-04-22 14:56:40,366 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/726c1db47ea84a3ea0366da7256fc021 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/726c1db47ea84a3ea0366da7256fc021 2023-04-22 14:56:40,367 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/f1db94c379ac490da9ce12fd9f8a123d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/B/f1db94c379ac490da9ce12fd9f8a123d 2023-04-22 14:56:40,369 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d4d9ff47c654432dbc3236438a90f236 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/d4d9ff47c654432dbc3236438a90f236 2023-04-22 14:56:40,370 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f156643cd8c0482183a513e1c89a1340 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/C/f156643cd8c0482183a513e1c89a1340 2023-04-22 14:56:40,372 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits/658.seqid to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb/recovered.edits/658.seqid 2023-04-22 14:56:40,373 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/.tmp/data/default/TestAcidGuarantees/89f2897412e14388617caefccca798eb 2023-04-22 14:56:40,373 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-22 14:56:40,373 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:56:40,374 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-22 14:56:40,376 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422035166b0ff844a26839eebe334686060 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422035166b0ff844a26839eebe334686060 2023-04-22 14:56:40,376 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042227205013a8834411bfc0448e71959e57 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042227205013a8834411bfc0448e71959e57 2023-04-22 14:56:40,377 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223b617084f0694ff7872a36b94b72032a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304223b617084f0694ff7872a36b94b72032a 2023-04-22 14:56:40,378 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224d52a5938d964145a9243bbf65a06dd4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224d52a5938d964145a9243bbf65a06dd4 2023-04-22 14:56:40,379 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224eea908598154d07af0b39535f6c2ad8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304224eea908598154d07af0b39535f6c2ad8 2023-04-22 14:56:40,379 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422515d083be6e9453a93b1900f473891aa to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422515d083be6e9453a93b1900f473891aa 2023-04-22 14:56:40,380 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256809a7888774577a9f6c1833cd30c1d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042256809a7888774577a9f6c1833cd30c1d 2023-04-22 14:56:40,381 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225fdd43a9d2d44eeeba32e5c969a29e30 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304225fdd43a9d2d44eeeba32e5c969a29e30 2023-04-22 14:56:40,382 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422616953fefb554c84b2cd93ee4b31258d to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422616953fefb554c84b2cd93ee4b31258d 2023-04-22 14:56:40,383 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042274f3ba8228d845c997b218e2de461bda to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042274f3ba8228d845c997b218e2de461bda 2023-04-22 14:56:40,383 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042279ee8238a0634cdf96deb3144089a93a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042279ee8238a0634cdf96deb3144089a93a 2023-04-22 14:56:40,384 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f93064e162b448fb2df2e1c193002f7 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304227f93064e162b448fb2df2e1c193002f7 2023-04-22 14:56:40,385 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042286ab6e58b91948e9b5ea1a1caab3c19a to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042286ab6e58b91948e9b5ea1a1caab3c19a 2023-04-22 14:56:40,386 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042291afed5bebfa4ec18a0b7198c8396ef4 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042291afed5bebfa4ec18a0b7198c8396ef4 2023-04-22 14:56:40,387 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a0b151c0cd6740c59f51f70ed700cff8 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422a0b151c0cd6740c59f51f70ed700cff8 2023-04-22 14:56:40,387 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b949184a509d4321b5665de476a5019c to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422b949184a509d4321b5665de476a5019c 2023-04-22 14:56:40,388 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bc5389713e7549438e38e14b3fb17c46 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422bc5389713e7549438e38e14b3fb17c46 2023-04-22 14:56:40,389 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422be1e64a37a374a638a06253438ad7f59 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422be1e64a37a374a638a06253438ad7f59 2023-04-22 14:56:40,390 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9c640a5c48c408faca0591ae9985d86 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422c9c640a5c48c408faca0591ae9985d86 2023-04-22 14:56:40,391 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d016633edc264187a5649238533afd87 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d016633edc264187a5649238533afd87 2023-04-22 14:56:40,392 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d17695d93b684dfab62d49c3632a2042 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422d17695d93b684dfab62d49c3632a2042 2023-04-22 14:56:40,392 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2bbe1b1614a47c29413d7030cb86894 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e2bbe1b1614a47c29413d7030cb86894 2023-04-22 14:56:40,393 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e43afd12c27945fdbdceb685ac13f998 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422e43afd12c27945fdbdceb685ac13f998 2023-04-22 14:56:40,394 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ef6e997927d64982a0d2c1b03b6cc100 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422ef6e997927d64982a0d2c1b03b6cc100 2023-04-22 14:56:40,395 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f81f25567175457fa3f77d0df955fce6 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f81f25567175457fa3f77d0df955fce6 2023-04-22 14:56:40,396 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f82f1dc8eaaa418399381f6f65c6e5d2 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422f82f1dc8eaaa418399381f6f65c6e5d2 2023-04-22 14:56:40,397 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fabe015e5a88474eae5642db562ee6b5 to hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230422fabe015e5a88474eae5642db562ee6b5 2023-04-22 14:56:40,397 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-22 14:56:40,399 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,400 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-22 14:56:40,402 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-22 14:56:40,402 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,402 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-22 14:56:40,402 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175400402"}]},"ts":"9223372036854775807"} 2023-04-22 14:56:40,404 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:56:40,404 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 89f2897412e14388617caefccca798eb, NAME => 'TestAcidGuarantees,,1682175365925.89f2897412e14388617caefccca798eb.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:56:40,404 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-22 14:56:40,404 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175400404"}]},"ts":"9223372036854775807"} 2023-04-22 14:56:40,405 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-22 14:56:40,407 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-22 14:56:40,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-04-22 14:56:40,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37547] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-22 14:56:40,457 INFO [Listener at localhost/45141] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-22 14:56:40,472 INFO [Listener at localhost/45141] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 229), OpenFileDescriptor=504 (was 591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=345 (was 385), ProcessCount=170 (was 170), AvailableMemoryMB=12840 (was 12704) - AvailableMemoryMB LEAK? - 2023-04-22 14:56:40,473 INFO [Listener at localhost/45141] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-22 14:56:40,473 INFO [Listener at localhost/45141] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:56:40,473 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:51853 2023-04-22 14:56:40,473 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:40,473 DEBUG [Listener at localhost/45141] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-22 14:56:40,473 DEBUG [Listener at localhost/45141] util.JVMClusterUtil(257): Found active master hash=185435837, stopped=false 2023-04-22 14:56:40,473 INFO [Listener at localhost/45141] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:56:40,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:56:40,476 INFO [Listener at localhost/45141] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:56:40,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:56:40,476 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:56:40,476 DEBUG [Listener at localhost/45141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2869d048 to 127.0.0.1:51853 2023-04-22 14:56:40,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:56:40,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:56:40,477 DEBUG [Listener at localhost/45141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:40,477 INFO [Listener at localhost/45141] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,32935,1682175218140' ***** 2023-04-22 14:56:40,477 INFO [Listener at localhost/45141] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-22 14:56:40,477 INFO [RS:0;jenkins-hbase4:32935] regionserver.HeapMemoryManager(220): Stopping 2023-04-22 14:56:40,478 INFO [RS:0;jenkins-hbase4:32935] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-22 14:56:40,478 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-22 14:56:40,478 INFO [RS:0;jenkins-hbase4:32935] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-22 14:56:40,478 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(3304): Received CLOSE for 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:56:40,478 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:40,478 DEBUG [RS:0;jenkins-hbase4:32935] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15cedc5d to 127.0.0.1:51853 2023-04-22 14:56:40,478 DEBUG [RS:0;jenkins-hbase4:32935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:40,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8ad7b375dc1e6947072b970f80229aca, disabling compactions & flushes 2023-04-22 14:56:40,479 INFO [RS:0;jenkins-hbase4:32935] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-22 14:56:40,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:56:40,479 INFO [RS:0;jenkins-hbase4:32935] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-22 14:56:40,479 INFO [RS:0;jenkins-hbase4:32935] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-22 14:56:40,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:56:40,479 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-22 14:56:40,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. after waiting 0 ms 2023-04-22 14:56:40,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:56:40,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8ad7b375dc1e6947072b970f80229aca 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-22 14:56:40,479 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-22 14:56:40,480 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 8ad7b375dc1e6947072b970f80229aca=hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca.} 2023-04-22 14:56:40,480 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 14:56:40,480 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 14:56:40,480 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 14:56:40,480 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-22 14:56:40,480 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 14:56:40,480 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-22 14:56:40,482 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1505): Waiting on 1588230740, 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:56:40,482 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:56:40,486 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-22 14:56:40,486 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-22 14:56:40,682 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1505): Waiting on 1588230740, 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:56:40,882 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1505): Waiting on 1588230740, 8ad7b375dc1e6947072b970f80229aca 2023-04-22 14:56:40,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/.tmp/info/84cf44f14e184a52abe6f76ecd890f82 2023-04-22 14:56:40,888 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/info/4f7c779ad2044e38844da3aedb8616ac 2023-04-22 14:56:40,892 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4f7c779ad2044e38844da3aedb8616ac 2023-04-22 14:56:40,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/.tmp/info/84cf44f14e184a52abe6f76ecd890f82 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/info/84cf44f14e184a52abe6f76ecd890f82 2023-04-22 14:56:40,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/info/84cf44f14e184a52abe6f76ecd890f82, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:56:40,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 8ad7b375dc1e6947072b970f80229aca in 418ms, sequenceid=6, compaction requested=false 2023-04-22 14:56:40,901 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/rep_barrier/0caa3ec111aa4c39b2394d156aba2cf6 2023-04-22 14:56:40,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/namespace/8ad7b375dc1e6947072b970f80229aca/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-22 14:56:40,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:56:40,904 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0caa3ec111aa4c39b2394d156aba2cf6 2023-04-22 14:56:40,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8ad7b375dc1e6947072b970f80229aca: 2023-04-22 14:56:40,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682175220542.8ad7b375dc1e6947072b970f80229aca. 2023-04-22 14:56:41,083 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 14:56:41,283 DEBUG [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 14:56:41,310 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/table/1c5b21d2ea13458e8ddcbbef39ae973e 2023-04-22 14:56:41,313 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1c5b21d2ea13458e8ddcbbef39ae973e 2023-04-22 14:56:41,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/info/4f7c779ad2044e38844da3aedb8616ac as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/info/4f7c779ad2044e38844da3aedb8616ac 2023-04-22 14:56:41,317 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4f7c779ad2044e38844da3aedb8616ac 2023-04-22 14:56:41,317 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/info/4f7c779ad2044e38844da3aedb8616ac, entries=22, sequenceid=93, filesize=7.1 K 2023-04-22 14:56:41,318 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/rep_barrier/0caa3ec111aa4c39b2394d156aba2cf6 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/rep_barrier/0caa3ec111aa4c39b2394d156aba2cf6 2023-04-22 14:56:41,321 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0caa3ec111aa4c39b2394d156aba2cf6 2023-04-22 14:56:41,321 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/rep_barrier/0caa3ec111aa4c39b2394d156aba2cf6, entries=6, sequenceid=93, filesize=5.5 K 2023-04-22 14:56:41,321 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/.tmp/table/1c5b21d2ea13458e8ddcbbef39ae973e as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/table/1c5b21d2ea13458e8ddcbbef39ae973e 2023-04-22 14:56:41,324 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1c5b21d2ea13458e8ddcbbef39ae973e 2023-04-22 14:56:41,324 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/table/1c5b21d2ea13458e8ddcbbef39ae973e, entries=9, sequenceid=93, filesize=5.5 K 2023-04-22 14:56:41,325 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 845ms, sequenceid=93, compaction requested=false 2023-04-22 14:56:41,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-22 14:56:41,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 14:56:41,333 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 14:56:41,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 14:56:41,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-22 14:56:41,483 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,32935,1682175218140; all regions closed. 2023-04-22 14:56:41,491 DEBUG [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/oldWALs 2023-04-22 14:56:41,491 INFO [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C32935%2C1682175218140.meta:.meta(num 1682175220247) 2023-04-22 14:56:41,496 DEBUG [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/oldWALs 2023-04-22 14:56:41,496 INFO [RS:0;jenkins-hbase4:32935] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C32935%2C1682175218140:(num 1682175219656) 2023-04-22 14:56:41,496 DEBUG [RS:0;jenkins-hbase4:32935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:41,496 INFO [RS:0;jenkins-hbase4:32935] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:56:41,496 INFO [RS:0;jenkins-hbase4:32935] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-22 14:56:41,497 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:56:41,501 INFO [RS:0;jenkins-hbase4:32935] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:32935 2023-04-22 14:56:41,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,32935,1682175218140 2023-04-22 14:56:41,506 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:56:41,506 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@696834d2 rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2163] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,507 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:56:41,507 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@31d1c8ca rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2163] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,508 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,32935,1682175218140] 2023-04-22 14:56:41,509 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,32935,1682175218140; numProcessing=1 2023-04-22 14:56:41,510 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,32935,1682175218140 already deleted, retry=false 2023-04-22 14:56:41,510 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,32935,1682175218140 expired; onlineServers=0 2023-04-22 14:56:41,510 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37547,1682175216913' ***** 2023-04-22 14:56:41,510 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-22 14:56:41,510 DEBUG [M:0;jenkins-hbase4:37547] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a308c86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:56:41,510 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37547,1682175216913 2023-04-22 14:56:41,510 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37547,1682175216913; all regions closed. 2023-04-22 14:56:41,510 DEBUG [M:0;jenkins-hbase4:37547] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:56:41,511 DEBUG [M:0;jenkins-hbase4:37547] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-22 14:56:41,511 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-22 14:56:41,511 DEBUG [M:0;jenkins-hbase4:37547] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-22 14:56:41,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219317] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219317,5,FailOnTimeoutGroup] 2023-04-22 14:56:41,511 INFO [M:0;jenkins-hbase4:37547] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-22 14:56:41,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219329] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219329,5,FailOnTimeoutGroup] 2023-04-22 14:56:41,511 INFO [M:0;jenkins-hbase4:37547] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-22 14:56:41,511 INFO [M:0;jenkins-hbase4:37547] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-22 14:56:41,512 DEBUG [M:0;jenkins-hbase4:37547] master.HMaster(1512): Stopping service threads 2023-04-22 14:56:41,512 INFO [M:0;jenkins-hbase4:37547] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-22 14:56:41,512 ERROR [M:0;jenkins-hbase4:37547] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-22 14:56:41,513 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-22 14:56:41,513 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:56:41,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:56:41,513 INFO [M:0;jenkins-hbase4:37547] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:56:41,513 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-22 14:56:41,514 DEBUG [M:0;jenkins-hbase4:37547] zookeeper.ZKUtil(398): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-22 14:56:41,514 WARN [M:0;jenkins-hbase4:37547] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-22 14:56:41,514 INFO [M:0;jenkins-hbase4:37547] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:56:41,514 INFO [M:0;jenkins-hbase4:37547] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-22 14:56:41,515 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 14:56:41,515 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:56:41,515 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:56:41,515 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 14:56:41,515 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:56:41,515 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.83 KB heapSize=393.71 KB 2023-04-22 14:56:41,525 INFO [M:0;jenkins-hbase4:37547] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.83 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/87b6b1302daa49759a17bb8bd2d85ad5 2023-04-22 14:56:41,530 INFO [M:0;jenkins-hbase4:37547] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 87b6b1302daa49759a17bb8bd2d85ad5 2023-04-22 14:56:41,531 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/87b6b1302daa49759a17bb8bd2d85ad5 as hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/87b6b1302daa49759a17bb8bd2d85ad5 2023-04-22 14:56:41,535 INFO [M:0;jenkins-hbase4:37547] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 87b6b1302daa49759a17bb8bd2d85ad5 2023-04-22 14:56:41,535 INFO [M:0;jenkins-hbase4:37547] regionserver.HStore(1080): Added hdfs://localhost:40923/user/jenkins/test-data/e268e561-8c89-31ad-9b0b-f55e7e5ec48c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/87b6b1302daa49759a17bb8bd2d85ad5, entries=65, sequenceid=579, filesize=24.7 K 2023-04-22 14:56:41,536 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegion(2948): Finished flush of dataSize ~344.83 KB/353111, heapSize ~393.70 KB/403144, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=579, compaction requested=false 2023-04-22 14:56:41,537 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:56:41,537 DEBUG [M:0;jenkins-hbase4:37547] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:56:41,540 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:56:41,540 INFO [M:0;jenkins-hbase4:37547] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-22 14:56:41,540 INFO [M:0;jenkins-hbase4:37547] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37547 2023-04-22 14:56:41,542 DEBUG [M:0;jenkins-hbase4:37547] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37547,1682175216913 already deleted, retry=false 2023-04-22 14:56:41,610 INFO [RS:0;jenkins-hbase4:32935] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,32935,1682175218140; zookeeper connection closed. 2023-04-22 14:56:41,610 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:56:41,610 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@49d91c02 rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2163] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,610 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@79de6730] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@79de6730 2023-04-22 14:56:41,611 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): regionserver:32935-0x101ac5cb37f0001, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:56:41,611 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@77d2c7fb rejected from java.util.concurrent.ThreadPoolExecutor@6d57ad27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2163] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,611 INFO [Listener at localhost/45141] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-22 14:56:41,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:56:41,710 INFO [M:0;jenkins-hbase4:37547] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37547,1682175216913; zookeeper connection closed. 2023-04-22 14:56:41,710 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c79b001 rejected from java.util.concurrent.ThreadPoolExecutor@1723ed0f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3620] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,710 DEBUG [Listener at localhost/45141-EventThread] zookeeper.ZKWatcher(600): master:37547-0x101ac5cb37f0000, quorum=127.0.0.1:51853, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:56:41,710 ERROR [Listener at localhost/45141-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@657cb60f rejected from java.util.concurrent.ThreadPoolExecutor@1723ed0f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3620] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:56:41,712 WARN [Listener at localhost/45141] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-22 14:56:41,716 INFO [Listener at localhost/45141] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:56:41,824 WARN [BP-354848111-172.31.14.131-1682175213564 heartbeating to localhost/127.0.0.1:40923] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-22 14:56:41,825 WARN [BP-354848111-172.31.14.131-1682175213564 heartbeating to localhost/127.0.0.1:40923] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-354848111-172.31.14.131-1682175213564 (Datanode Uuid ea2500fb-007a-47ad-b293-b124c6c82704) service to localhost/127.0.0.1:40923 2023-04-22 14:56:41,827 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a/dfs/data/data1/current/BP-354848111-172.31.14.131-1682175213564] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:56:41,827 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fb3d5d7b-b539-3dfd-9edf-427586316db1/cluster_cbc046c3-4ade-2d6c-a957-847d1ba3e12a/dfs/data/data2/current/BP-354848111-172.31.14.131-1682175213564] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:56:41,864 INFO [Listener at localhost/45141] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:56:42,031 INFO [Listener at localhost/45141] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-22 14:56:42,199 INFO [Listener at localhost/45141] hbase.HBaseTestingUtility(1293): Minicluster is down